Uploading CSV with 1 Million rows to backend within 10 seconds

yobulk 646 days ago. link 1 point ▲ ▼

Inserting large records from a spreadsheet (CSV file) to Backend is a very common and hard engineering problem.

Modern data stack ETL tools may solve it ..But

- You are a web developer (javascript)person..:)
- You don't want to invest money on an ETL tool.
- You are not a data engineer who create and maintain data pipeline.

Please read this blog to understand how YoBulk is solving this in a most frugal(limited CPU and RAM) way.

https://github.com/yobulkdev/yobulkdev/tree/main/pages/api/upload

tracker1 646 days ago. link 1 point ▲ ▼

I'm not sure why OpenAI is a requirement for this kind of work?  Also, how many requests (at $.09 each) are there for a CSV with a million rows?

yobulk 646 days ago. link 1 point ▲ ▼

There is no Open AI requirement for this.It's purely solved using nodejs streams.

kali 644 days ago. link 1 point ▲ ▼

There are various mentions on the project's readme https://github.com/yobulkdev/yobulkdev about certain "Open AI features" and the need for an Open AI key. This is probably what tracker1 was asking about it.