General Programming Discussion

7814 readers

5 users here now

A general programming discussion community.

Rules:

founded 5 years ago

MODERATORS

-6

submitted 11 months ago by cuenca@lemm.ee to c/programming@lemmy.ml

10 comments fedilink hide all child comments

One needs to send 1 mln HTTP requests concurrently, in batches, and read the responses. No more than 100 requests at a time.

Which way will it be better, recommended, idiomatic?

Send 100 ones, wait for them to finish, send another 100, wait for them to finish… and so on
Send 100 ones. As a a request among the 100 finishes, add a new one into the pool. “Done - add a new one. Done - add a new one”. As a stream.

you are viewing a single comment's thread
view the rest of the comments

[–] vmaziman@lemm.ee 2 points 11 months ago* (last edited 11 months ago) (1 children)

Maybe producer consumer?

Producer spits out all the messages to send out onto a message queue, fifo or whatever suits u.

Parrallelizable consumers (think deployed containers) listen to queue and execute request, get response and save it

Scale consumer count up or down as you need to deal with ratelimits

[–] cuenca@lemm.ee -3 points 11 months ago

What question have you answered?