Imported from pinboard.in/u:timburks
Links tagged 'avro'
Switching to Protobuf (from Avro) on Kafka 🚀
https://www.clearstreet.io/news/blog/switching-to-protobuf-from-avro-on-kafka
I'm a little surprised to see a story of an Avro to Protobuf migration - I thought teams were moving in the other direction - but this might be driven more by other existing uses of Protobuf in the same organization.
Indeed, this is part of a bigger story about both developer and system performance.
Our inflection point was developer struggle. Our developers were dealing with eight or nine formats every day and writing hundreds of lines of code just to convert from one format to another. If we didn’t standardize, we’d spent 40% to 50% of the code on converters alone.
Protobuf started to become the dominant data format at Clear Street. Simultaneously, we hit issues with Avro and our change data captures (CDCs), which took data from our database and moved it onto Kafka using Debezium. On top of that, we had problems with schema migrations using Avro that could potentially lead to massive rollbacks.
Also, check this out!
Much of what we’re developing is open source. For example, all the changes we made to KafkaUI are on Clear Street’s GitHub. We are in the process of making a push request back to the original authors explaining how we’d added Protobuf support to KafkaUI.
How to Document Event-Driven API Message Payloads 🚀
https://apichangelog.substack.com/p/how-to-document-event-driven-api
In summary, you can safely use Avro Schemas to define your event-driven API message payloads.
⟪ recent ⟫ advice aftershokz agents agi ai alienhominid alltheplaces android apimanagement apis apisyouwonthate appengine apple art auden automation avro badshah beastieboys benfolds bios blogging boba bobdylan books breaches breakfast brevity brunopedro bsky buf bullshit cacm cameronblevins capitalism changesets chatgpt cherylwaters christophkern cli cloud cloudrun cncf coffee commenting community companies concerts conferences connect cplusplus css dart dartmouth dashboards data databases datalakes datapoisoning debugging defunkt design devex devsite diet dirtywave documentation easteregg eda editions editors edm eks empire endpoints engineering envoy events faith family finch flaxseed frost fruit ftc gallbladder games gateway gateways gcp geekbench geo girard github gloo go google googlemaps gorilla gregorymone grpc grpcweb hacking health healthchecks heartworms help heresy hichord history homelab http hype hypebusting iceberg ideas imgoing india innerengineering inonshkedy integrations interviews iusethis jamesmurphy java jennifergovola jokes json juliaangwin k8s kafka kagi kaitenzushi keithharing kelseyhightower kentstate kexp kiosks kubernetes law lcdsoundsystem licenses linkblogs llms localfirst locations lucagalente lyrics m8 malloryhaigh martinkleppman matduggan materialdesign mccarthy meetups meridethwhittaker meta microsoft middleware minipcs minneapolis minsky museum music nat networking nginx npr nutrition nyt openapi opensource openstreetmap operators oreilly otobokebeaver overture pancakes performances pescatarian peterdenning pharisees pinboard pinkpantheress platformcon platformengineering platforms podcasts poetry portland portugaltheman postgis postideas privacy production productreviews programming prost protobuf protocolbuffers protos pubsub python quality ransomware raphaelpinson recipes repos rss rtree rubrik rust saas sabotage sadhguru santaclarauniversity score scrapers scu sdks seahorse search security sfmoma signal snl snowflake software songs soup spotify spotifyengineering sqlite startups steelydan storage strawberries styleguides sudorandom super73 sushi synthesizer synthhistory teams teensy tiles timbowmanjr timburks tinydesk toddlyons tonic trackers travel turing unkey usps vanta vegan via:license victortangermann videos vulnerability walking web webarchive webinars weezer wikimedia williamdalrymple wix workflows workouts yoga youtube zed zombiezen