maartenballiauw,
@maartenballiauw@mastodon.online avatar

Experiment... As the "search engine" behind a 5000-records auto complete API is an LIKE query. Loading 60000 more records into that table made the API endpoint slow (6 seconds to show autocomplete!)

How would you improve this autocomplete?

jan,
@jan@social.dytrych.cloud avatar
khalidabuhakmeh,
@khalidabuhakmeh@mastodon.social avatar

@jan @maartenballiauw You win a prize! That’s the answer Maarten was fishing for. 😅

Wicked Tuna Fishing GIF by National Geographic Channel

maartenballiauw,
@maartenballiauw@mastodon.online avatar

@khalidabuhakmeh @jan Am I this transparent? :D

khalidabuhakmeh,
@khalidabuhakmeh@mastodon.social avatar

@maartenballiauw @jan Select * from Obvious

khalidabuhakmeh,
@khalidabuhakmeh@mastodon.social avatar

@maartenballiauw Can I do anything to solve the issue?

maartenballiauw,
@maartenballiauw@mastodon.online avatar

@khalidabuhakmeh Anything. What would be your first attempt to improve it?

khalidabuhakmeh,
@khalidabuhakmeh@mastodon.social avatar

@maartenballiauw Honestly, I wouldn’t mess around with anything but a proper search engine like ElasticSearch or Azure Search in your case.

While it can “sorta work” within database search indexes, the control over how search occurs is limited.

I’d just rewrite the endpoint (since 60K is nothing to ES).

b4ux1t3,
@b4ux1t3@hachyderm.io avatar

@khalidabuhakmeh @maartenballiauw This is why I recommended just sticking it in a traditional search/index structure. 60k is so few records that, if I don't expect it to grow that much in the future, I wouldn't even lean on ES. I'd spend more time configuring ES than implementing a custom search index. Heh.

khalidabuhakmeh,
@khalidabuhakmeh@mastodon.social avatar

@b4ux1t3 @maartenballiauw Please elaborate by what you mean “traditional search/index” structure.

Is this SQL Server full-text search, or PostgreSQL full text search?

maartenballiauw,
@maartenballiauw@mastodon.online avatar

@b4ux1t3 @khalidabuhakmeh Let's say the records don't grow or change that often. Would that change your approach?

khalidabuhakmeh,
@khalidabuhakmeh@mastodon.social avatar

@maartenballiauw @b4ux1t3 🤪 Then I have a crazy idea for you, buddy!

Build and deploy a SQLite database with all the records and Full-Text Search at deployment time, and then use that. No extra service costs.

https://www.pythonlore.com/advanced-sqlite3-full-text-search-implementation/

b4ux1t3,
@b4ux1t3@hachyderm.io avatar

@khalidabuhakmeh @maartenballiauw SQLite is cheating, it does what I was telling you to do without any actual work >:(

(so many problems solved with SQLite)

khalidabuhakmeh,
@khalidabuhakmeh@mastodon.social avatar
  • All
  • Subscribed
  • Moderated
  • Favorites
  • dotnet
  • rosin
  • cubers
  • osvaldo12
  • DreamBathrooms
  • khanakhh
  • magazineikmin
  • thenastyranch
  • Youngstown
  • everett
  • InstantRegret
  • slotface
  • ngwrru68w68
  • kavyap
  • tester
  • provamag3
  • cisconetworking
  • GTA5RPClips
  • Durango
  • mdbf
  • ethstaker
  • modclub
  • tacticalgear
  • normalnudes
  • megavids
  • Leos
  • anitta
  • JUstTest
  • lostlight
  • All magazines