Ramkumar Kamaraj @_PerfEngineer

Fun fact: OpenAI handles 800 million users on ChatGPT with just one PostgreSQL primary and 50 read replicas 🤯 Today, OpenAI published an engineering blog explaining how they scaled their Postgres setup to support a massive 800 million users using a single primary and 50 multi-region replicas. They dive into details around their scaling approach, the PgBouncer proxy, cache locking, and cascading read replicas. It is genuinely neat and impressive. I just published a video on my YouTube channel where I dissect the blog and break down the nuances. Give it a watch - it is short and fun.

@ganeshsonawane Turn every trip into a dream journey with the Frido Travel Neck Pillow! ✈️✨ Engineered for comfort, it’s your passport to blissful naps on the go. 💤 Wherever you wander, Frido's got your back (and neck)! #FridoTravelNeckPillow #TravelSmart

Sovereign Gold Bonds are supposed to be very safe investments. But, my experience with them is really poor My SGBs matured on Aug 5, 2024. But, funds yet to be credited. Issuing bank @centralbank_in has no clue where is that money; zero clarity even after 2 months @RBI @RBIsays

I’m gonna give 10 random people one frido car neck rest each for free who repost this, comment frido and follow me. I’ll pick and announce the winners in 72 hours!! myfrido.com/products/frido… #ContestAlert #Giveaway #frido

@JioCare , My JIO Fiber connection is not working for last 3 days, Issue booked via your App is not taken care your service engineer as per the slot assigned. No calls from them, not picking any of the calls.

@RadnuS81 Seeing the route suggestion as im planning to visit Chidambaram from Chennai this week. still confused if i need to take 1) Vikravandi-Panruti-Vadalur- Sethiathope - Chidambaram 2) Ulundurpet-Vridhachalam-Sethiathope - Chidambaram If any one recently travelled, please suggest

@DealsDhamaka @YogiTweetz What is the renewal fee for MRCC from 2nd year onwards?

Top 7 Ways to 10x Your API Performance 1. Pagination This is a common optimization when the size of the result is large. The results are streaming back to the client to improve the service responsiveness. 2. Asynchronous Logging Synchronous logging deals with the disk for every call and can slow down the system. Asynchronous logging sends logs to a lock-free buffer first and immediately returns. The logs will be flushed to the disk periodically. This significantly reduces the I/O overhead. 3. Caching We can cache frequently accessed data into a cache. The client can query the cache first instead of visiting the database directly. If there is a cache miss, the client can query from the database. Caches like Redis store data in memory, so the data access is much faster than the database. 4. Payload Compression The requests and responses can be compressed using gzip etc so that the transmitted data size is much smaller. This speeds up the upload and download. 5. Connection Pool When accessing resources, we often need to load data from the database. Opening the closing db connections add significant overhead. So we should connect to the db via a pool of open connections. The connection pool is responsible for managing the connection lifecycle. On 6 and 7, we created a video about this topic and included 2 more ways to improve performance. Watch and subscribe here: lnkd.in/e5JJfbcs

Refresher on 𝗥𝗲𝗮𝗱𝗶𝗻𝗴 𝗗𝗮𝘁𝗮 from 𝗞𝗮𝗳𝗸𝗮. 𝗥𝗲𝗺𝗶𝗻𝗱𝗲𝗿: ➡️ Clients writing to Kafka are called 𝗣𝗿𝗼𝗱𝘂𝗰𝗲𝗿𝘀. ➡️ Clients reading the Data are called 𝗖𝗼𝗻𝘀𝘂𝗺𝗲𝗿𝘀. ➡️ Data is written into 𝗧𝗼𝗽𝗶𝗰𝘀 that can be compared to tables in Databases. ➡️ Messages sent to 𝗧𝗼𝗽𝗶𝗰𝘀 are called 𝗥𝗲𝗰𝗼𝗿𝗱𝘀. ➡️ 𝗧𝗼𝗽𝗶𝗰𝘀 are composed of 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝘀. ➡️ Each 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻 is a combination of and behaves as a write ahead log. ➡️ Data is written to the end of the 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻. ➡️ Each 𝗥𝗲𝗰𝗼𝗿𝗱 has an 𝗢𝗳𝗳𝘀𝗲𝘁 assigned to it which denotes its order in the 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻. ➡️ 𝗢𝗳𝗳𝘀𝗲𝘁𝘀 start at 0 and increment by 1 sequentially. 𝗥𝗲𝗮𝗱𝗶𝗻𝗴 𝗗𝗮𝘁𝗮: ➡️ Data is read sequentially per partition. ➡️ 𝗜𝗻𝗶𝘁𝗶𝗮𝗹 𝗥𝗲𝗮𝗱 𝗣𝗼𝘀𝗶𝘁𝗶𝗼𝗻 can be set either to earliest or latest. ➡️ Earliest position initiates the consumer at offset 0 or the earliest available due to retention rules of the 𝗧𝗼𝗽𝗶𝗰 (more about this in later episodes). ➡️ Latest position initiates the consumer at the end of a 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻 - no 𝗥𝗲𝗰𝗼𝗿𝗱𝘀 will be read initially and the 𝗖𝗼𝗻𝘀𝘂𝗺𝗲𝗿 will wait for new data to be written. ➡️ You could codify your consumers independently, but almost always the preferred way is to use 𝗖𝗼𝗻𝘀𝘂𝗺𝗲𝗿 𝗚𝗿𝗼𝘂𝗽𝘀. 𝗖𝗼𝗻𝘀𝘂𝗺𝗲𝗿 𝗚𝗿𝗼𝘂𝗽𝘀: ➡️ 𝗖𝗼𝗻𝘀𝘂𝗺𝗲𝗿 𝗚𝗿𝗼𝘂𝗽 is a logical collection of clients that read a Kafka Topic and share the state. ➡️ Groups of consumers are identified by the 𝗴𝗿𝗼𝘂𝗽_𝗶𝗱 parameter. ➡️ 𝗦𝘁𝗮𝘁𝗲 is defined by the offsets that every Partition in the Topic is being consumed at. ➡️ State of Consumer Groups is written by the Broker (more about this in later episodes) to an internal Kafka Topic named __𝗰𝗼𝗻𝘀𝘂𝗺𝗲𝗿_𝗼𝗳𝗳𝘀𝗲𝘁𝘀. ➡️ There can be multiple Consumer Groups reading the same Kafka Topic having their own independent States. ➡️ Only one Consumer per Consumer Group can be reading a Partition at a single point in time. 𝗖𝗼𝗻𝘀𝘂𝗺𝗲𝗿 𝗚𝗿𝗼𝘂𝗽 𝗧𝗶𝗽𝘀: ❗️ If you have a Prime number of Partitions in the Topic  - you will always have at least one Consumer per Consumer Group consuming less Partitions than others unless number of Consumers equals number of Partitions. ✅ If you want an odd number of Partitions - set it to a 𝗺𝘂𝗹𝘁𝗶𝗽𝗹𝗲 𝗼𝗳 𝗣𝗿𝗶𝗺𝗲 𝗡𝘂𝗺𝗯𝗲𝗿. ❗️ If you have more Consumers in the Consumer Group then there are Partitions in the Topic - some of the Consumers will be Idle. ✅ Make your Topics large enough or have less Consumers per Consumer Group. -------- Follow me to upskill in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space. Also hit 🔔to stay notified about new content. 𝗗𝗼𝗻’𝘁 𝗳𝗼𝗿𝗴𝗲𝘁 𝘁𝗼 𝗹𝗶𝗸𝗲 💙, 𝘀𝗵𝗮𝗿𝗲 𝗮𝗻𝗱 𝗰𝗼𝗺𝗺𝗲𝗻𝘁! Join a growing community of Data Professionals by subscribing to my 𝗡𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿.

@DealsDhamaka Hi. I'm not able to apply this card through your referral link.

Why I don't use Scrum to manage my Remote Teams? TL;DR: It adds at least 8 hours of meetings per Sprint. That's 2 full days of time wasted, per team member, per month! This is what I do instead:

performance.shopify.com/blogs/blog/imp…

Create a web performance story in 3 easy (okay, easy-ish🙈) steps courtesy of @WebTwitr! Check it out on the @Spotify Engineering Blog: engineering.atspotify.com/2022/09/from-d…

10 Key Takeaways From New Relic’s 2022 Observability Forecast Report forbes.com/sites/janakira… via @forbes

📚. IT'S HERE! We have been diligently getting this ready for back to school! We're proud to announce that we have a ✨FREE✨ course for ALL wanting to learn. Led by @scottjehl, ⚡️ Lightning-Fast Web Performance will fill all your needs. RT! #webperf blog.webpagetest.org/posts/announci…

@akaasi All three sir, first people, then infra and last policing

🍱 Ever been to a restaurant, ordered & received appetizers, main course, drinks, dessert all at once? That’s the case when you 'PRELOAD' too many resources. @tkadlec discusses the perils of such practices, and how they impact paint metrics. #webperf blog.webpagetest.org/posts/removing…

If I could combine Three Articles that I re-read recently (along with a few visuals) into One Thing that could make your head explode and your self-reflections & career conversations to grow exponentially in effectiveness, I would pick the following three: 🧵🧵🧵

keyboard_arrow_left Previous keyboard_arrow_right Next

Check out this incredible avalanche footage from the Tian Shan mountains in Kyrgyzstan. Luckily, everyone survived. Account: cutt.ly/fLkOldS

@puthagappuzhu @Im_Revathy Aswins before perambalur is good for morning breakfast on Madurai - Chennai route with good parking.