Vibecoding

Admin

Administrator
Moderator
Messages
4,391
#1
I recently got into vibecoding because i wanted to get an accurate score distribution for my favorite poker variant.

It's actually a really nice game once you understand the rules so i wanted more people to be able to try it. I recently expanding the evaluator to cover all variants of it and so far i have not found it to make a single error.

evaluator-NLP8.png


It actually started with me simply wanting to get an accurate distribution over the scores and i did get what's hopefully going to be an accurate distribution via chatGPT and claude (i asked bot to make a sim so i could compare results and hand evaluations). The distribution i got ended up being nicer than i expected it to be.

score-distribution-100M.png


Later i made a version where you play against 3 bots but that did require me to provide a lot of guidance due to LLMs having a poor understanding of poker theory. I did however not see any need to make the bots super-good since the goal was to let people easily try out the game and for that purpose they are already good enough even if they are fairly easy to get an edge against.

https://nlp8.vintologi.eu/

Main thing that remains is to get a working sim for trying out the profitability of various starting hands (when the mediocre bots play them) but i am already quite satisfied with what i have been able to achieve just using free AI (i did however have to jump a bit between models due to free plans running out and due to some models not being able to do what i asked of them). I think i will need to redo the post-flop model from stratch since the model made by the AI was quite bad (that became very clear when i looked at the actual code).

People like to hate on AI but there is no way i would have been able to make this without it without spending weeks/months learning programming (especially not considering i had to both make C++ and HTML/javastript versions.

I did do some manual editing of the code on my own (such as to put links where i wanted them and to adjust MDF weights for the defense against all-in shoves).

One thing that did help was to have a detailed PDF explaining the poker variant (including examples) since that helped the AI models understand what i meant and also gave it examples to test it against. Still it did get some stuff wrong which i ended up having to ask it to correct later.
 

Admin

Administrator
Moderator
Messages
4,391
#2
AI has a tendency to take do a shoddy job to produce output instead of doing things properly
You end up having to direct it over and over again to actually do things in a more proper manner.

So if you rely on AI and do not check what's it actually doing you end up with loads of shitty code and you get stuck with very sup-optimal software where there is no good way to improve it further because the entire things is broken and dysfunctional to the core (to the point where you end up having to start from scratch.

Often when you tell AI to do something it will not actually do as instructed and instead do something significantly worse and you might not actually notice that unless you actually examine the code to see what it actually did (the software may seem to work ok at first but then you will notice that things don't quite add up).

I earlier got the impression that chatGPT had done some good empirical work to develop a decent preflop evaluator only to later discover that the model it made was awful and then when i tried having it improved the model it really didn't want to do things properly and it just kept doing stuff wrong. I asked it to change the model to make sure it aligned with earlier all-in equity measured (against 2 random hands) but it didn't actually align it properly and admitted that after i told it to examine the model.
 

Admin

Administrator
Moderator
Messages
4,391
#3
Very often when an AI claims to have fixed some issue the issue isn't actually fixed
They make you think the problem is solved but then you run into the same or similar bug again or you discover that the supposed fix introduced more problems than it solved.

This is in line with the general theme when using LLMs for coding. You have to constantly watch what they are doing and direct them, especially if you are not using a top tier model. It's like having a dishonest employee that will try to make you think they are working for you but that will constantly cheat and do dishonest things in general behind your back.

I had chatGPT claim to have fixed a problem 5 times only to then find out that the problem wasn't actually fixed. That's pretty bad if you trust the AI to code something of actual importance (you might think it fixed some problem only to later discover it just made things worse).
 

Admin

Administrator
Moderator
Messages
4,391
#4
I was eventually able to get an exact score distribution for my poker variant (i think)
I did verify the frequency at 13 and 12 points myself using libreoffice calc.

ChatGPT did get the same distribution down to 10 points but then my free plan ran out and i wasn't able to continue. Later i asked claude and got matching answers down to 9 points. I then asked claude to make me a brute-forced evaluator but seems like it didn't work properly (i did abort before it could finish due to initial results for when you just select 8 cards out of 52 not matching what results claude gave earlier).

NLP8-score-distribution.png


Which ended up matching what i later got when running an exact hand frequency counter from claude.

Claude-NLP8-exact-score-distribution.png


Here the claude version first scored every single 8-card combo and then used that to get the exact frequency when 8 cards out of 11 are selected. The chatGPT version instead brute-forced it by looking at every single 11 card combo directly and picking the best possible score for each (when selecting 8 cards).
 

Admin

Administrator
Moderator
Messages
4,391
#5
My poker variant is now playable online thanks to vibecoding (and chinese AI)
I naively thought it wouldn't take that much more effort to make a website to play with others when i already had a site where you could play with a bot. Then when i actually tried asking AI models to help me with it basically all failed and the only one who got to a decent start was claude.

Claude did however add a bot and then just ignored me when i asked about it (even though i stated there should be no bots).

But free claude is very limited i ended up having to switch to other models that weren't as good for most of the actual work to make the website more functional and eventually it got quite good.

https://nlp8a.vintologi.eu

I did end up having to manually editing the HTML directly once due to deepseek not following basic maths and thus breaking the website for a screen width range. I also manually edited the HTML to add a table over the score frequencies (which i got earlier) and add 2 links.

My chatGPT accounts got limited and i didn't feel like making a third one so i ended up using deepseek for most of the work despite it being overall worse than chatGPT and claude (not that any model i tried was particularly great for this). I once noticed that it stopped working completely after i updated the code for the node.js server and then it turned out that there were a lot of dhost instances running (due to restarts not properly killing the old process) which caused me to be limited by my hosting provider. Deepseek recommended running "pkill node" in a terminal via cpanel which did get rid of those problematic ghost processes (i do know how that command works, it may also kill some other processes as well but i never noticed any issue from it).

There were many occations where deepseek needed a lot of attempts before it actually fixed a problem, it would tell me the issue had been fixed but it wasn't fixed and i sometimes had to tell it to start over. Deepseek even changed back from the CSS i had fixed to broken CSS ignoring the new fixed version i had uploaded. When you don't pay with money you often end up paying with your time instead.
 
Top