[00:00:00] Speaker A: This episode brought to you by Studio Network Solutions Media teams have enough things to worry about.
Storage shouldn't be one of them. That's where Studio Network Solutions comes in. SNS makes your shared storage, media management and cloud workflows easy so you can focus on what you do best creating.
See how SNS can help your
[email protected].
[00:00:28] Speaker B: You know, in the world of live production there are two types of people. Those who think they can rely on Sketchy Hotel, Wi Fi and those who use LiveU. See, LiveU is the industry leader in bonded cellular technology, meaning you can go live from just about anywhere. Stadiums, newsrooms, a moving car, probably even the moon. But we'll let NASA test that one. From broadcasters to YouTubers, LiveU makes it easy to deliver high quality, low latency video with without the stress. No more crossed fingers, no more laggy feeds. Just rock solid reliable live production. So if you're tired of watching your stream buffer while your viewers drop off, check out LiveU. Because in live production there are no second chances.
Have you ever watched a production and thought wow, that lighting is perfect. But then the host stares into the void like they forgot how to read.
Yeah, that's because they didn't use icann. See, ICANN makes top tier studio lighting prompters and gear that keep productions looking and running smooth. So whether you need buttery soft LED lights or rock solid teleprompters so you don't forget your lines or just pro level gear that won't let you down, Ikan has you covered. So if you want your production to look like a million bucks without spending a million bucks, check out Ikann because great lighting and a good prompter can make anyone look like a Pro.
[00:01:55] Speaker C: 32 Red 1.
[00:02:01] Speaker D: Hey everyone, thanks for tuning in to another episode of Broadcast to post. I'm Steve Dupay and this week's episode is going to be packed with some deep insights, field tested lessons and a few strong opinions on ST2110 workflows. Now, the timing for this episode couldn't be better. We've just wrapped up KeyCode Media's largest 2110 project to date and at KMGH Denver 7. And let me tell you, this was a fully ground up all IP all ST2110 broadcast facility Greenfield installs are one thing, an all IP all day every day greenfield facility. Whole different animal. And we learned a lot about timing, about interop, about what works and what bites you in the ass if you don't plan for it up front. But more importantly, so did our client. Things like ptp, clock issues, network topology surprises, vendor gear, not talking to each other the way it should. We've seen it all in this install now. Side note, if you're the kind of person who loves technical write ups and lessons learned docs, good news. Our marketing team is working on a full Denver 7 case study which will land on our LinkedIn soon. Make sure to follow us there if you want the nerdy deep dive.
Back to the podcast though, here's the plan.
First, we're doing a rapid fire on top ST2110 products out of NAB 2025 and a little from last year too.
Then I'll be talking with Nick Kumar, our new VP of Broadcast Engineering here at Keycode. You may know him from his work at ASG on the massive PAC12ST2110 build. And finally, we're bringing in some true heavyweights. Robert Erickson from Tag Video Systems and Ryan Morris from Arista Networks. And a real conversation about what's actually happening in the field when it comes to interoperability, latency and getting teams ready for ip.
All right, let's kick off with the Rapid Fire update. Here are your ST2110 headlines straight from NAB 2025, what's hot, what's shipping, and what you need to be paying attention to. Ross Video dropped the Carbonite hypermax, a hyperconverged beast, switching, routing, multi viewing, compositing all in one box. Plus their Ultrix line just leveled up with an expanded IP routing and hybrid SDI P workflows. Grass Valley introduced the ACE 3901 Gateway, which can handle a whopping 384 SDI to IP transitions in four ruins. Also worth noting, the Kaleido IPX multiviewer is optimized for both ST2110 and NDI environments. Tag Media Systems is going big supporting uncompressed and compressed ST2110, MPEG, TS, SRT, you name it, real time transcoding, advanced signal analysis. Their monitoring capabilities have widely expanded and their platform offers deep probing, real time visualization and advanced analysis of video and audio signals across various workflows including live production, playout and OTT delivery. Arista Networks offered Advanced guidance on ST2110 design focused on multicast operations software defined architecture and precision timing. They also introduced their IP Patch Bay, a software defined feature that can analyze run diagnostics on multicast IP traffic from any network port on your media fabric. Netgear showed off their new M4350 series switches PTP ready and built for AV over IP. Their new Engage 2.0 platform also streamlined setup in a big way.
Imagine Communications is backed with updates to the Selenium Network processor ready for uhd, HDR and hybrid cloud workflows. Lavo continues to integrate Edge and build upon their home platform which is a management platform for IP based infrastructures and broadcast signal processing. At ebirds they got the NATX16 compact IP router perfect for Remy and Flypak, and big enhancements to Magnum OS for bandwidth and extreme control. AI Media builds upon their AI based closed captioning offering with Lexi which provides high accuracy real time captioning including live events, broadcasts and educational environments. Lexie also featured live language translation capabilities and captioning transcriptions the market is moving fast, but you don't just need gear, you need strategy. And that's where the next part of this episode comes in 32 red one so you might have heard, we made a bit of a splash recently by bringing in Nick Kumar as our new VP of Broadcast Engineering. Nick's been around the block. He led the ST2110 install at the PAC12 network while at our friendly competitor ASG and he knows where the bodies are buried when it comes to IP builds. We're going to shake things off with Nick now, get a little background, hear his take on where things are going and later he'll join our panel. Nick, welcome to the team and welcome to your first broadcast to post episode as an official Key Code Media crew member.
[00:06:25] Speaker A: Thank you Steve, Glad to be here. I'm a broadcast engineer by trade and I have been in the systems integration business since 2006. I have a Bachelor's degree in Electrical Engineering and a Master's degree in Engineering Management and my involvement with projects has ranged from live production facilities, network origination facilities, major market TV stations and sports networks.
I started off when our customers at the time were modernizing and transitioning their systems from ST to HD and I got involved with a lot of these types of infrastructure upgrade projects and that is when our paths also crossed. Steve, if you remember, my priority has always been customer focused and help them develop solutions based on their specific needs and timeline and of course budget. Oftentimes I work in close collaboration with the customer so much that I am often regarded as an extension to their engineering leadership team.
I like to take a methodical approach when dealing with projects and try to keep all aspects of the project thoroughly documented. I also believe in adequate and advanced planning so that we have proper implemented plans, commissioning plans and checklists tailored to the customer's specific workflows internally within the various systems and integration organizations that I've also that I've worked. I like to focus on continual improvements in our internal processes to maximize efficiency. This can include basic things like managing internal resources or even automating various aspects of how we do systems design and over the last five or six years I have focused a lot on SMPT ST21 fence system and infrastructure deployments and advanced media networking workflows for several customers.
[00:08:04] Speaker D: It's fantastic background Nick, you've led several ST2110 builds over the last six years and a few different indicators. What was your role in those major installs and what significant things did you learn from them?
[00:08:18] Speaker A: My role has been primarily that of a principal engineer where I would not only help architect the overall SMT ST2100 system and networking blueprints and generate project cost estimations, but also manage high level conceptual designs for advanced media networking deployments.
One thing that I quickly learned along the way that even as an integrator, the focus should not just be defining and designing the physical layer, but to perhaps transcend to the upper layers of the OSI or TCP IP model. This would include creating more detailed logical workflow type diagrams to represent the true design intent in an SDI system. You can easily assess what the system entails just by following the design as the flow is very unidirectional. But with SD2110 implementations, although the physical layer design may be more simplified now as you have endpoints connected to a network fabric, the workflow representation of what is happening inside the system needs to be abstracted and documented separately in these major installs. I've also been the boots on the ground personnel and leading the overall project installation and systems commissioning team along the way, all the way to user acceptance, testing and client handoff. The customer list with my experience in 2110 deployments include some major market O and O stations in the D.C. and New York area, various corporate broadcast customers, network origination and play out facilities, and of course more recently PAC12 enterprises with their new studio build out.
[00:09:45] Speaker D: Tremendous background, great insight and a toolkit that is just par excellence. So great to have you on board and we look forward to lots of success with you in the years to come. Thanks Nick, thank you very much.
[00:09:57] Speaker A: Glad to be here.
[00:10:03] Speaker D: All right, let's shift to our main discussion. The reason we put this episode together is to help engineers, system designers and operations leaders Understand the real world realities of moving to ST2110, what works, what breaks, and how to get ahead of the curve.
So to help us unpack all that, we've got Robert Erickson from tag. He's been one of the clearest voices advocating for practical ST2110 solutions. And Ryan Morris from Arista, who knows more about networking design than most of us care to admit. Let's dive in. Robert Ryan, great to have you both here. And Nick, feel free to chime in anytime as well. Let's get right into it. So Robert, how did we get here?
[00:10:42] Speaker C: So it's actually a bit of a long story and is completely driven by what the industry needed to do at the time.
So if you look before 2110, of course everything was based off of SDI. And SDI always had this very fixed relationship. One input into one router, one router into one output, so forth. It's a very physical relationship to cable signal, cable so forth. Right now, as the need, as the customers demand for media increased, we had to grow infrastructures to handle more sources and more destinations. Well, SDI routers are inherently RF routers. Like a 1.5 gigabit per second digital signal is actually 750 MHz signal on a router, or a 3 gigabit per second signal is actually a 1.5 GHz signal on a router. So routers are inherently RF and there's incredible challenges to grow those routers and sizes. So you'd always think that like 1000 inputs and 1000 outputs is huge and massive. Well, is starting in about the 2010, 2012 time period, customers had infrastructures that needed 2000, 3000, 4000 sources and destinations.
[00:11:51] Speaker D: Right.
[00:11:52] Speaker C: And that RF base that the SDI routers, being RF based, just couldn't scale. It couldn't grow because you have to divide and multiply rf. And again, the physics get really, really messy. So the industry had to find a different way. And of course, the first step of that was we need a way to convert the SDI video into ip.
And so there was already an existing standard in the 2010, 1213, and that was SMPTE2022 6. MT2022 6 was incredibly simple. It very literally took the video with the embedded audio that was on the SDI signal and just encapsulated it and sent it to a switch, which is kind of cool. But it had some limitations because if you wanted to modify the audio or you want to draft the audio in that you had to send the video and audio to a processor board to de embed the audio, kind of like we had to an SDI and then do all the processing and put it all back together. So there had to be a better way. So the industry created a standard called TR03, which eventually became SMPTE 2110. And so 2110 took that video that was initially in the SDI stream and just made it its own essence SMPT 2110 20. It took the audio that was embedded in that SDI stream and made it its own essence SM2021 1030 and all the rest of this stuff, the closed captions, AFD flags, things like that, were also created in their own essence, which is MT21 1040. So you can kind of see the industry really needed the ability to grow, but they had to create the standards. They had to create the infrastructure to support it. And that's where 2110 came in. So once we remove that physical interface, once we remove that one to one relationship of signal to cable and got rid of the RF out of the equation, and then we started bringing in things from Arista and other switch manufacturers to be able to handle all that processing as data, as a function of data and as a function of bandwidth instead of a function of rf.
That now allows us to grow our networks to almost any size as long as you have the functional bandwidth to support it.
[00:13:59] Speaker D: The other thing it did is it changed the cabling infrastructure that we would use as well. Right. And that's kind of a big part of this. The amount of physical SDI cable that's just unidirectional within a plant.
[00:14:12] Speaker C: You know, it's actually crazy to think about this, but one of the latest jobs that we've done with arista was all 400 gigabit interconnects. And just think about the reduction of cabling and the efficiency of your. Your system wiring, you know, just on that link alone. You know, it's also. That's been another part that was fun too. It's like when we first launched 2110, we were 10 gig at the edge and 40 gig at the core. And then for the past couple years, we were doing 25 the Edge and now at 100 gig at the core. Now almost everything we're doing is 100 gig at the Edge, 400 gig at the core. So we've just seen this absolutely incredible rate of change and of increasing the amount of bandwidth almost every couple years. I'm sure Ryan can speak to this more on what's trending on that. But that's something that's been very advantageous in terms of wiring and also for efficiency and infrastructure.
[00:15:05] Speaker D: And I think that kind of pretends that the adoption rate now of 2110 is growing because of the benefits that are starting to really show up and the advantages that people have in their systems. Would you agree with?
[00:15:20] Speaker C: Yeah, that's exactly it. And also the price is coming down so much too.
You know, it used to be when we first did 21:10, it was only the customers that had the absolute requirement to go IP to do it, and it was enormously expensive.
And now we're finding nowadays that 2110 systems, especially given the cost of switches being down, the fact that most devices now are native 2110, you know, you don't have to do a lot more conversion anymore, especially on greenfield sites.
That has driven the cost of 2110 networks down to almost parity for most workflows. I mean, there's a few areas where, if you're talking like, you know, 16 and 16 out or so forth, that SDI still may be cheaper, but for almost any modern infrastructure, the cost of 2110 is getting close to parity in terms of the equipment. Don't get me wrong, there's still some complexities in terms of staffing and training that I know we're going to talk about further into this, but we're getting to the point now that it is almost cost parity of building out 2110 as it is on SDI in most situations.
[00:16:16] Speaker D: Right. Hey, Brian, let's move to you. What are the migration challenges that you've seen in adopting 2110 and what things have made it easier as we've gotten into it here in the last few years?
[00:16:29] Speaker E: Yeah, thanks, Steve.
I think some of the challenges have always been around the configuration side of the, of the network. You know, it is a bit of an, you know, there's a learning curve, right, of learning CLI commands on how to properly configure ptp, how to properly configure multicast routing and thereby unicast routing. There's been a lot of, you know, there were challenges with, with that. Again, it's a, it's a, it's a steep learning curve, but people have, you know, people have mastered that. You know, the configurations are taking a lot less time and they're becoming more accurate. And something that we're really working on with, you know, with various teams here at Arista are our automation techniques.
So, you know, we have our automation techniques Like Arista, validated designs, avd.
And that's really beneficial for any of these kinds of large scale or even medium scale SMPTE 2110 or media systems where we can have the configurations all templatized and built before a single switch even arrives at the customer site.
So you don't need to wait until the, you know, switch is powered on, it's physically connected, you have management connection to it before you build the configurations. Of course, if you will of course need all the cabling done, you will need to reach management, the manager networks, the multicast, you will of course need all of that.
But to actually get the configuration work completed, that can be done well, well, well in advance.
And then once the switches are essentially booted up, the configurations are just beamed down to the switches. You know, entire 2110 solutions right now are built without anyone connecting into the CLI of the, of the switches.
[00:18:19] Speaker D: Wow.
[00:18:19] Speaker E: So that's really helping because again you're, your configurations are becoming more accurate, they're being done quicker. And of course talking to sis, what do SIS really like? Documentation that is all built for you. With these kinds of solutions, you want to know what's connected to what.
That's all part of the whole automation journey. So that's really simplifying the overall deployment process. Now in 2025 and even a bit.
[00:18:43] Speaker D: Earlier, I would think that that also makes it possible to then troubleshoot quicker the issues that may arise from various manufacturers interpretation of sdp, of ptp, of the overall architecture, so that you're spending your time not architecting the system in situ, but you're actually troubleshooting the issues that are going to plague a plant no matter when you get the thing set up. Is that correct?
[00:19:11] Speaker E: Yeah. Like the automation techniques won't necessarily assist with why is the device joining the stream. But what it will do is provision the network correctly so you don't have to go back and look at the configurations and be kind of uncertain as to whether or not my multicast routing is built correctly. So you can focus on the other, on the endpoint devices, for example, or on the control system. We're on an orchestration system. You don't have to worry about the actual configurations of the network devices themselves.
It lets you kind of focus your attention elsewhere, I would say.
[00:19:49] Speaker D: Right. It kind of feels like that would also make it possible to have automated the best practices kind of approach embedded in that initial configuration so that you're less likely to make mistakes with where you assign your multicast, where you're assigning your ptp, how you're setting some of those things up, your priorities and those kinds of things. Am I right?
[00:20:11] Speaker E: No, absolutely. So we take best practices that we've learned on our various deployments and we do apply those in our automation techniques. So there are very specific standards for assigning PTP messaging rates to devices. Now not everybody will of course abide by those best practices. Some devices like prefer higher rate of announce messages, for example. Of course you can change that. But you are absolutely correct. We take those best practices, we apply those into our automation techniques and with a minor configuration you can say, I want all of my endpoint devices to be on the AES R16, 2016 profile. So absolutely, yes, that is a key component of this. Applying the best practices, the validated best practices and building the templates and deploying those, you know, en masse.
[00:21:06] Speaker D: Yeah, reducing risks of errors during integration or operation. That's a huge, huge add. Kudos to Arista for taking that initiative to do that.
What other hurdles do you see still out there on legacy facilities, on phased rollouts, those types of things? And where do you see the biggest bottlenecks today?
[00:21:26] Speaker E: So I think we're, I think some of the bottlenecks really do come on the educational side. We'll be getting into this more, I think later on. But you know, there are lots of courses out there. There are lots of YouTube videos of people do, you know, producing their best practices, but just watching the videos and just taking a course like it's great to get your knowledge up, but you have to kind of get the hands on experiences, you know as well you have to be able to know how to go into a switch or go into a monitoring system and, and know what to look for, what is the bandwidth? Here are my devices reporting the right IGMP reports.
Are my senders, for example, not making it to a receiver because a TTL is one and the packets are just dying once they get routed.
So I think learning those kinds of techniques for effective troubleshooting is something that I think as an industry, you know, we're, we're all, we're all striving to, to improve that.
[00:22:26] Speaker D: Yeah, I would agree. I'd like to take all the notes down of all the problems that I've seen in, in the 2110 installations I've done and, and have them captured and, and automate them in some of the design setups going forward. So. Yeah, good stuff.
[00:22:42] Speaker A: Brian mentioned about, you know, proper documentation.
So the standards have matured, right? I mean the standards were ratified back in 2017. And so there's a lot of lessons that we've learned along the way, and being able to take all those lessons learned and apply them as best practices and being able to forecast some potential issues beforehand and applying best practices in your documentation.
So proper planning goes a long way in a successful project, and not just necessarily from a configuration standpoint only, but also proper commissioning best practices, proper testing tools.
When I first started, I didn't necessarily have a whole lot of network configuration background, but I did have a pretty solid networking understanding. But over the years, as I learned, I logged and I recorded a lot of the different commands, networking commands at a CLI level that I would use, and I would then apply them on future projects.
So having a good understanding, being able to properly plan beforehand is absolutely key to making sure the project is successful.
[00:23:50] Speaker D: That kind of leads back to, in terms of inventor interoperability, if we have those best practices in place, there's a need to make sure that things interoperate correctly. Where do we stand with that in the industry and how can we track that better in our documentation so they can make sure that things work well when we first fire them up?
[00:24:13] Speaker A: Sure, yeah. So like I mentioned, I mean the standards were ratified in 2017. Right. So the application and adoption of these standards into technologies offered by different manufacturers has only matured. Right. So in the beginning there were a litany of issues with vendor interoperability across different platforms. But over the last seven to eight years, we have seen a great deal of maturity of adoption with these standards and systems integration of best of breed SD2110 systems.
There's always going to be real life challenges like you mentioned, and these usually come up with new product releases or a manufacturer that's introducing SD2110 capable products.
But it is important as consultants and systems integrators to vet these products either through exhaustive research or proof of concepts.
Audio processing channel mapping used to be a huge challenge with 211030 in the earlier days of the standards adoption. And it can still be a challenge, especially if you have multiple flavors of production audio formats in your plan, such as 2110 30, A67, Dante, Livewire, what have you. And some of these formats are interoperable and can coexist with others provided that the networking is set up adequately for it. But in a lot of real life scenarios and real life implementations, we do see a separation of audio networks connected together with some kind of bridge device.
Or there might be challenges in the way networks need to be designed based on client requirements. For example, your master control playout may want to be on a separate network architecture, and live production may want to be on another. So bridging the two architectures will require some proper planning during the design phase and very close communication with the core vendors, especially the networking vendor.
PTP issues can still arise if the network is not configured properly or if you have a misbehaving endpoint. But by and large, with the maturing of the standards and its wider adoption, and of course, lessons learned along the way, a lot of these issues need not necessarily be a problem anymore. Of course, there may be new variations to these issues that you uncover in a build specific, certain endpoints, sure.
[00:26:31] Speaker D: And that kind of ties into other areas that may have some. Some issues. Right now there's been a lot of, I don't want to say problems with NMOs, but a little bit of confusion and a little bit of challenge. A few challenges, let's say, with that, with NMOs and getting everybody to behave the same. Can you talk a little bit about what your experience has been with that and how you resolve some of those issues?
[00:26:57] Speaker A: Yes, of course. Nmos. So nmos, it stands for Network Media Object Specifications. And there are a set of specifications which has been authored by the Advanced Media Workflow association.
And there absolutely have been improvements to NMOS support as a lot of the major vendors in the market and emerging new vendors, they're increasingly developing systems that are now JTNM tested and certified.
So JT and M certifications, by the way, represent tests and measures the ability of a device and their conformance to the SD2110 set of standards and specifications.
But and there is growing maturity with wider implementation of, of course, ISO 4, which specifies device discovery and registration, and ISO 5, which defines the communication between the broadcast controller and endpoints to manage connections between them.
Additionally, there are other NMOS implementations, such as ISO 8, which represents audio channel mapping ISO 9, which deals with automation and rudimentary device setups ISO 10, which adds security and authorization layer to NMOS approaches. And these are all specs that are published and gaining maturity in wider deployments. However, we still do find vendors that only partially implement specs or sometimes will use proprietary extensions.
So I have even seen some vendors that only support, for example, NMOS ISO 4 and not ISO 5, which would make it difficult to use this particular endpoint as a receiver because I am not able to patch or do any ISO 5 SDP patching into its connection API receptacle.
So some broadcast Controllers made by certain manufacturers would work seamlessly using their proprietary protocols with other processing endpoints made by that same manufacturer. But in order to communicate and interoperate with other third party devices, NMOS becomes an option that is almost treated like the lowest common denominator.
But I must add that most of the major broadcast controllers in the market today have a solid report card with their NMOS compliance to work with third party endpoints. But then again, there's still endpoints that may have compliance issues. And then of course comes the fun part of developing some workarounds to make these systems functional.
[00:29:23] Speaker D: And I think that's where the expertise of a particular integrator can really be advantageous is that experience of dealing with those kinds of problems with various vendors, various products, help us to inform the customer a little bit better about which products are going to work well together, which ones may present a little bit more of a challenge and, and take a little bit more time to integrate correctly.
[00:29:43] Speaker A: Absolutely.
[00:29:44] Speaker D: And I think that also provides a great feedback to the manufacturers as well that getting your products working correctly and uniformly with the specifications will ensure a broader usage and greater success with other people's products in the system.
[00:30:02] Speaker E: And Steve, if I can add a bit to that, you know, while I'm seeing NMOS in pretty much every single deployment, you know, it's there and it works great. One thing I do want to just, you know, talk, you know, talk to, you know, the various customers about is the various testing. Like while NMOS might be working fantastic in a steady state system, always test out network failures, switch reboots, those kinds of things. Different devices have different behaviors from an NMOS perspective. Perspective, some are out of band, some are in band. Understanding those kinds of nuances and how the controllers keep on communicating with those devices during not just the best of times when everything is up, but also the worst of times when you have to upgrade a switch or power down a switch is paramount to ensuring that you maintain as high of an uptime as possible.
[00:30:55] Speaker D: Robert, let's jump back to you for a minute.
TAG has developed a lot of interesting tools in their product. And so I think you're very much aware of these kinds of issues.
Where do we sit with latency, with lip sync and those kinds of things in 2110 based systems as compared to SDI's? And where are we making progress and where does it still create a production risk?
[00:31:19] Speaker C: Yeah, so monitoring, it's already been brought up a little bit, those monitoring tools is something I think has actually advanced a lot in the industry. Over the past couple of years, when we first started doing 2110, a lot of the testing tools were still based off of SDI technology.
We're going to route one signal into one analyzer into one scope, and that one scope is going to do all the analyzing for that one signal. And that's all you can possibly do.
That has been around for a very long time. And to be quite honest, that methodology does work.
Tag has developed the ability to monitor and analyze every Signal in the 21:10 flow all the time, as long as it hits the platform. We can monitor the ptp, we can monitor the PTP drift, we can monitor the rtp, we can monitor the RTP drift, we can do a waveform vector analysis on every Input on every 2110 flow in the entire system. And that's just how the industry and also how software based tools have advanced things. You know, this isn't something to brag about, Tag. This is something more about talking about the dynamic nature of our industry.
This is talking about how when we first did IP, the industry largely applied 20 applied SDI style workflows to IP environments. And now that we're having true software defined infrastructures and the ability for media processing and media monitoring to become pieces of software have allowed us to create these new workflow and more important workflow tools because of it. And there's also, at least on the TAG side, I'm not trying to make this a tag promotion here, but they do pay my salary, so I got to put a little bit in here.
But if you look at, I like the monitoring the multi viewing of a broadcast facility, every critical feed is going to hit that multi viewing platform because you're going to want to be able to visualize it. You're going to want to be able to see every critical feedback.
Well, if you're already decoding the video and the audio and the metadata to be able to visualize it, why don't you just monitor and probe it at the same time? It doesn't add any overhead to the system, it doesn't add any additional multicast joints, it doesn't add any additional bandwidth to the system.
It just becomes one of those things where since you're already subscribing to the flows anyway, let's do a very rich analysis of that.
To be very clear, there is other manufacturers, not just in the monitor and probing domain, but that are using that same mentality of if we're going to subscribe to this flow, why don't we just do multiple transform functions on A flow because we can. And again, this is the part I'm really trying to harp on a little bit here just because I'm getting kind of passionate about it is now that 2110 is mature, 2110 is ubiquitous.
And the industry is starting to create these new workflows. And creating new tools to create new workflows allow us to do things that we can never do in the SDI domain.
And it's the same with.
You're talking about avoid lip sync issues.
It's pretty rare we run into them on 2110. I would say it's very rare we do run into them at 2110. But every once in a while you do have a bad actor on the system that will, you know, whether it's a it needs a reboot or whether something's gone wrong with its stack that audio and video do tend to drift.
They do have tools available. Tag makes tools available where we can tell you if lip sync drifts as it goes through a production chain.
But we also look at most of 2110 environments are largely in a LAN environment. And so sometimes when you worry about AV lip sync drift, it is partially because of a piece of processing gear that is either delaying or forwarding audio and video compared to the other one.
Or in the WAN environment, you get into some tricky things where there's always a possibility because the audio and the video are actually separates multicast IP addresses. There is an extraordinarily rare but occasionally annoying chance where they can be routed through two different routes with different timing thereof.
In the LAN environment, that's really rare. Also in the LAN environment, usually the paths are so short that any pathological differences or buffering differences between the audio and video is going to be almost perceptually.
You won't notice that the test gear won't even notice it. But now that 2110 is starting to push over a WAN again, this is us starting to adopt workflows that we can do. You know, 2110 was traditionally always land based. I'm going to put my switches in my racks. I'm going to have my gear in this facility and this is where I'm going to do my production.
Now that we're Starting to see 21:10 push into cloud environments and push into Remy environments, especially with the addition of compression on 2110. So SMPTE 2110, 22, there's the availability now to do this in a remote environment.
There is the distinct possibility that the audio and video IP addresses multicast. In this case can take separate routes.
Now I'm sure Ryan's like, you can program the switches to not allow that to happen.
Yes there is. But people do forget to do that and sometimes things do happen that we can't perceive or can't expect. But AV Sync is rarely a problem, but can be, especially as we're pushing into WAN environments. But again, broadcast manufacturer equipment manufacturers are designing to have designed network tools that will automatically detect that AV Sync issues, whether it's on a LAN or a WAN and tell the engineers about it. It just comes with the maturation of the product and the maturation of the industry where now you no longer have to wait for it to happen and happen for a long time and wait for somebody to complain about it. We can tell you in real time if that's happened and tell your engineers exactly where to go to fix it.
[00:36:37] Speaker D: That's phenomenal. And I think that's one of the really cool things about being in that IP environment is the software solutions that are much more readily available than they could ever have been in a purpose built type of solution are going to advance our ability to test, monitor and correct things and in an advanced form rather than waiting until as you just said, the problem is complained about somebody's noticing and calling the helpline to say hey, this isn't working. Right.
[00:37:08] Speaker C: And I must say that's also in my opinion probably the biggest change that 2110's brought to though the broadcast environment isn't just the fact the original goal is we just need more sources, more destinations. That was the original goal. Right.
But the ability for us to manage these media workflows and do all the processing and media workflows now as a function of software because again the media is now a function of bandwidth, which means we can just wrap that into software based solutions. That's going to be the big change.
No longer having to have purpose filled hardware that does one job and that's all it possibly does. Those days are going to run out. It's going to become a pure software domain where the software is adaptable and more importantly as adaptable as the customers workflows require them to be.
[00:37:49] Speaker D: Yeah, I think we're going to touch on that a little bit later that that second wave as one manufacturer refers to it and with the what the EBU is doing to help make that reality in the near future.
Ryan, let's go back to you for a minute.
You touched on this a little bit before with when you were talking about the automated setups that you guys are putting together for configuring the network ahead of time. What would you say is the biggest skill gap engineering teams face as they move from baseband to network infrastructure? Where do they need.
How do we help people get to where they need to be so that their 2110 implementations, whether it's a system integrator, an operator, an engineer at a television station, or someone in the field, so they can be more successful with using these new tools.
[00:38:34] Speaker E: There are a couple of very key features in any SMD 2110 media workflow that really stand above the rest. And that's my opinion at least. There's PTP and then there's, you know, there's of course, multicast. I think having a really strong understanding of multicast, at least to be able to understand, like, why are signals dropping when they shouldn't be dropping? How do I know if someone's actually looking for a signal or if I'm using some kind of orchestration system?
How do I check the switch and how do I check the orchestration system itself to note to verify that the multicast flows have actually been programmed?
Those are, I think, the biggest gaps. And things are evolving, right? Like we used to have these systems that were exclusively IGMP managed. And a lot of people, including myself, focused a lot on dynamic multicast protocols, IGMP and pim, PIM sparse mode. Now that we're going to, now the systems are evolving, you know, we're not just having like one big modular chassis because, you know, where's the fun in that? You want a distributed workflow, you want to have a spine leaf system generally. And again, these types of spine leaf systems are great from a scalability perspective. You can expand east to west with your choice of switches, whether it's for supporting one gig devices, 10 gig devices, 25 gig devices, whatever it happens to be. Plus, you know, you don't need to have these spine switches be big modular boxes anymore. So kind of understanding the hardware updates and the workflows, but these kinds of actual workflows are changing how we actually deliver the multicast groups from the senders to the receivers. And again, because it's changing, it's hard to just. Sometimes it can be challenging to just keep up with these new solutions. And that's where these kinds of webinars really help out. Or talking to the network teams to see how are they treating these kinds of systems now, how are we routing the multicast and then in addition to that, what kind of monitoring solutions are available? So Robert was obviously just talking about monitoring the video and the audio integrity of the signals. That's absolutely critical, 100%, completely critical. But then there's also the network monitoring tools as well. Knowing are there discards on certain links, what is the bandwidth on certain links and getting that information in real time, whether that's from cloud vision, our monitoring system, or from another third party monitoring system that's consuming that same telemetry. I think a big gap is understanding how to use these monitoring tools that are available to better help with troubleshooting these systems.
[00:41:26] Speaker A: I also want to just add on the importance of testing tools and monitoring.
So there's also a lot of free product out there.
AMWA offers a free NMOS testing tool which creates a web service to test implementation of your NMOS APIs in your system and then retle. NMOS Explorer is also a really great tool for testing, for querying and ISO 5 connection management APIs. It's almost like a poor man's broadcast controller that could be used in a test or lab environment.
And then there's SD Poker as well that can test the legality of SDP files and check for syntax. I've seen SDP files that are all over the place and they just don't pass the legality when you run them through this CLI tool, which is STP Poker.
[00:42:11] Speaker D: Which takes me, Nick, to a question for you. We've got a lot of facilities out there that are SDI that are looking to go ip. We're talking with people every day about how to help them migrate, whether they need to start from scratch, do a greenfield or maintain what they've got and add to it in a hybrid approach.
Can you talk a little bit about where that is at, what best practices are helping to avoid, doubling up their cost, doubling up the effort, et cetera, and migrating to an IP based workflow.
[00:42:45] Speaker A: One big best practice is you want to consider a unified router control system, right? The last thing you basically need is a disparate control system for controlling your SDI endpoints, SDI IO, and then another one that controls your IP environment. A true hybrid routing system will have the capability to manage, you know, all your signal flows and I O holistically.
ROS's hyperconverged platform is an excellent example of, you know, this type of topology and you know, the routing of all the external 2110 signals to and from, you know, the frame will still require some kind of underlying networking infrastructure, but all the control of all the SDI and 2110 signals, it happens in a unified manner through BCS Other things to also consider is, especially when you have facilities that are trying to balance the cost of IP upgrades while they're maintaining SDI infrastructure, is you want to try to maybe break the project into manageable little pieces, right? So, so that it doesn't get too logistically challenging.
And the process is usually best handled by breaking it down by functionality or a subsystem by subsystem basis. And in typical builds like these, what we've seen is you would start off by building a hybrid routing environment where you may start off by architecting your networking topology separately, along with an overarching broadcast control system and perhaps even an SDN network control that would have more control mechanisms over your media fabric. And then of course, as you start to upgrade From SDI to 2110, you would develop a plan to allocate them to the new media fabric. And then some endpoints may require SDI to IP conversion with NCAPs or DCAPs. But one thing that often gets overlooked at the beginning of these types of exercises, especially perhaps maybe during the consulting phase, is the general consideration to what the new overall power requirement is going to be with an IP system footprint. Right? How does it affect my overall cooling in an equipment room? For example, do I have enough cooling to support both my IP infrastructure that I'm building out gradually and also support my current legacy SDI systems in tandem as I do my phased cutovers? Do I have enough cabling access to run all my new fiber optic cables for my IP conversion project, or do I need to install new type, you know, new fiber trays or conveyance, et cetera, et cetera? So these are some of the most, you know, some of the high level, you know, critical things, crucial things that you want to consider when you're building a hybrid environment.
[00:45:27] Speaker D: I think that's excellent. Quite a checklist there.
It also reminds me that years ago I had a kind of a saying that if you would move away from a control system for each individual product and move towards a system control type of approach.
So I think you're, you're right on the right on the money on that one.
As budgets tighten and we have greater requirements, do you think we're going to be able to be able to maintain stability in systems and provide adequate redundancy and failover and backup?
[00:46:07] Speaker A: I mean, we are, I mean, a lot of our customers are. And you know, one thing about this industry is that you get to work with, you know, and maybe you even get to work with a lot of talented and clever Engineers. Right. And a lot of these engineering teams are getting very creative, right, in meeting compliance and redundancy requirements.
[00:46:26] Speaker C: Right?
[00:46:26] Speaker A: So, you know, for example, redundancy is prioritized, right? Based on the criticality of the subsystem. It may not be a standard that is applied to your entire media topology, where, let's say on air critical systems may have redundancy designed in whether it's SD2022 7 based, seamless protection, switching redundancy, or actual redundant air chains, PTP Grandmaster redundancy would be warranted as well.
And then of course, you would probably want redundant SPGs and for good old blackbirds reference, for lesser critical paths such as maybe editing and graphics processing that's mostly internal to the plant, maybe those systems can be single threaded.
A lot of times this can lead to an asymmetrical red and blue network topology, but that is okay. I mean, that's something that is based on design, right? I mean, that's how we're designed, designing it as a means to perhaps value engineer the project. But the thing to perhaps, maybe watch out for is to make sure that you have a 20227 compliant device that is connected to an asymmetrical red and blue network, that the network hops it would take for the primary path and the network hops it would take for the blue path. There's not a huge path differential between the two streams.
And make sure that they're within the tolerance level of the 20227 specs right now. Granted, a lot of this is happening in a LAN environment and these thresholds are fairly forgiving in these type of environments, but it's something just to keep an eye on as well.
[00:48:11] Speaker D: That's excellent. This has been a masterclass. I'm having a hard time keeping up with you guys because there's so much good information coming here. Well, I appreciate everybody's comments today and your insight. This has been a tremendous learning experience for me and hopefully for all of those that participate in the podcast.
Robert Ryan, Nick, appreciate all of your input here and your insight and we'll move on to the question and answer period.
[00:48:36] Speaker C: 3, 2, Red, 1.
