Episode Transcript
[00:00:00] Speaker A: Foreign.
[00:00:05] Speaker B: Welcome to today's podcast. I'm Stevie Dupe, Senior Director of Broadcast Innovation with Key Code Media. Today we're diving into one of the biggest transformations shaping the broadcast infrastructure, the shift from baseband to IP, with a focus on the advanced SIFTE ST2110 standard. And who better to guide us through this journey than Andy Rylance, Director of engineering at Denver 7 News. Just over a year ago, Andy and his broadcast team made a groundbreaking move to adopt Synthe 2110, or Denver7's Broadcast Infrastructure Backbone, while moving their newsroom facility to a new location. Today we'll revisit the recently completed project to explore the lessons learned, the challenges they tackled and the benefits they're reaping from an IP based workflow. Our company, KeyCode Media, was selected as the broadcast system integrator, helping KMGH with design, vendor selection, installation and commissioning. And I personally was very involved in this project and can speak to the key takeaways we Learned from this 2110 deployment. We'll cover a lot of ground from the practical impact of this transition on daily operations to the latest advancements in 2110 and how it will impact broadcast design moving forward. So whether you're a broadcast engineer, technical director, or involved in broadcast infrastructure planning, this conversation promises actionable insights from navigating the future of broadcast. Thanks for joining us this morning, Andy. I know it's been a few months now since Denver7 has gone online with their full facility in News room upgrade to SMPTE 2110. Can you tell us a little bit about some of the biggest lessons you've learned from this implementation and how it's impacted your day to day operations?
[00:01:37] Speaker A: Yeah, absolutely Steve, and thanks for inviting me to the editing podcast. It's as you say. We went live back in July and it's our first month, maybe a little shaky, but we've been getting steadier and steadier and you know, we're in a great place and so happy with our decision to go in this direction.
Lessons learned I think the first real lesson is, you know, 21% is real. It is. The standard is very solid standard. It's complex. It's kind of scary at first with the number of standards, number of substandards, all which, you know, form a very key part of gelling the system together. But once you become familiar, it's, you know, it becomes almost a second nature.
It is, it's very completely, say, providing a lot of benefits, a lot of added functionality compared to previous standards, SDI and Such like we, you know, in the early days we had a little bit of concern about integration, just how solid would be. I think the 2110 standard at the time, at the time we made the decision was very well ratified. It was, you know, interoperability was out there, there was less of a concern. Nmos on the other hand was relatively new. It was already ratified 2019 and we a little concerned about, you know, whether vendors would be, would be fully there.
Another takeover. We find one of the kind of key goals and successes I would almost call it is that the, the systems to an operational viewpoint shouldn't appear any difference. The change from SDI to 2110, it's back end, it's all in the rack room, it's all in the equipment, in the switches and to an operator, to a studio director, a master control operator, it should make no real difference. And you know, choosing the right control and orchestration systems I think really helps to achieve that.
[00:03:45] Speaker B: What, what was critical with that, with that decision?
[00:03:50] Speaker A: We're undergoing a big change. I mean, you know, we moved from a facility that's been in place for 50 years, 50 or 60 years, and to you know, full new facilities, some new systems. We didn't, you know, there is investments to change with staff and we wanted to try to limit how much change we presented. Um, obviously we wanted to add functionality, wanted it to add capability, we wanted to enhance our resilience, but we didn't want it too much of a shock. We knew it would be a pretty tight turnover for our moving from our old facility to the new and we wanted to keep that in hand.
[00:04:30] Speaker B: Did it work out well?
Is your staff feeling comfortable with how they command and control the system today?
[00:04:37] Speaker A: Extremely well.
They, I think from a, an operations viewpoint, you know, we, we build a PCR new PCR a couple of years ago, three or four years ago, based on the same systems Automation Systems, Sony ELC at that time with the XVS7000 switcher, SSL sandbox for the previous. And we took the opportunity to, to learn, you know, what worked and what we could have done a bit better and what had better integration within the the 2110 world redid our PCR, redid the layouts, reworked how we do audio and the, the end of the day our studio staff and directors very positive feedback that we've received.
[00:05:23] Speaker B: Great. In terms of its transparency for operation, would you say it was, was there a lot of learning to do or a moderate amount, a little amount. Where would you say that part lands operationally?
[00:05:36] Speaker A: A small amount it would be pretty much the same as you know, any move from an up. We moved from Ignite, as I said previously, to eos, Sony, eoc. And so moving from a larval and sorry from an SSL and SDI based production control room to our new control room. It was, you know, it wasn't revolution. It was just an evolution. It was a step up from an engineering viewpoint, very, very different from an engineering viewpoint. Oh my God. It is really learning from the, from the ground up again.
And that was more I think in terms of gaining comfort with our technical staff to understand just what 2110 is, how it works and get over that initial hump again, the unknown, the big change.
Our staff, our technical staff took to it very, very quickly. You know, we, we had a, a few hands on sessions, we had some training, we had some familiarization. Getting hands on is what made the difference and what really helped to, to build that comfort zone.
[00:06:41] Speaker B: Yeah, you, you can learn a lot of theory but until your, your, your fingers are actually touching stuff, it's, it's kind of hard for that to, to cement in your mind.
Would, would you say that, that once you had operational control of the facility that that knowledge and capability increased pretty quickly or did you find that that was a struggle? I know you talked a little bit about it there, but I kind of want to get into that a little bit more.
[00:07:06] Speaker A: It did and it depends, you know, certain individuals, some individuals took to it faster and those who were very hands on at the beginning. I think one of the key aspects in gaining that comfort is having staff involved during the commissioning stages. During the commissioning, the testing and the rollout don't, you know, we did not want. Is one of the reasons that we, we like Kiko's approach is that they encouraged us to be a part of the project and an integral part of that, that project. So getting hands on and setting up devices, playing with us and troubleshooting it really gain my comfort was was there.
[00:07:46] Speaker B: Some specific training that you did in the IT domain to get you ready for it? Was there, was there a challenge for guys that maybe hadn't touched the IT portions of systems much before coming up to speed on it?
[00:08:00] Speaker A: Yeah, I mean there's the, you know, your standard network training. We went with Comptia. It's a more open training. It's not, you know, or in 20 specific vendors. So that was the intro to networking. Most of the guys had been trained and anybody knew joining the team, we encouraged to take part in that training. We had very specific arista Training we have an Arista switch stack for our media core. Rst2110 media core. A hugely important part of our operation and we've had a staff through training, Arista specific training. We also had most of the staff go through some's intro to SD2110 and don't be, don't be fooled. It's called intro. I think it's called name is intro to 21:10. But by God it's, it's deep. You get hard into, you know, at a very intimate level with the standards. And honestly a couple of our guys went through to completion and certification which was fantastic. Others went so far. We were happy with that. It was getting people immersed in that training and as understanding the core concepts which, which was the important part. I started the formal training of those in the early stages. To be honest with. We had several vendors come in and there was show and tell. We had larva come in and they've, you know, experience, a lot of experience in the standard. Same with imagine a lot of experience in these systems. We invited them come in. We had all of our engineers attend. They walk through what 2110 is, what it works, how it works, what it can do, what it can't do and how to prepare ourselves in readiness for the, for the transformation. And again, every one of those different approaches, but any, every one of those helpful in, you know, building the comfort, building the confidence of our team to attack these systems.
[00:10:03] Speaker B: Great. So a lot of early prep, a lot of detailed prep in terms of understanding how the systems would work, understanding the technology itself so that you were ready to take take command and control of it.
[00:10:15] Speaker A: Yeah, yeah, yeah.
[00:10:17] Speaker B: On the SDI side, the question I really want to ask is, you know, how much SDI did we end up with? How much IP based and what impact did you see that having on the facility?
[00:10:32] Speaker A: Yeah, I mean we had a specific budget to the project. We were hit pretty hard. We started planning for this, we budgeted this in 2019 and we all know what happened in the next couple of years. So our budget was somewhat impacted, but not to any point that we had to make a rapid change in direction. So we did reuse some of the equipment that we brought from our old facility.
Only if it was a couple, you know, had a relatively, it was relatively new and had a long lifespan ahead of it.
I think really the important part is gateways. SDI to 2110, 21 to 2 SDI gateways in the past have always kind of struck us as cost prohibitive. It's, it makes it very difficult for an expensive project if you are filling racks with gateways. Gateways have come down in cost. There weren't as I think by the time we started deplanning for this, it wasn't quite as scary as it first seen. We found in many cases. Well, not in many. In a couple of cases, it was actually lower cost for us to add a gateway to a device than it was to build in the 2110 to the device. It means you have an extra bit of coax in place, but it functions equally well. We try to keep that down to a minimum.
What we did, what we do have in place, we've, you know, the gateways work well. We had no real issues with, I would say and I'd add a proviso to that in making sure it's, you know, it's fully ratified and broadcast gateways. We did an issue some lower cost gateways later in the project and had to make a very quick U term.
[00:12:17] Speaker B: Yeah. So selecting the gear that is, it's fully baked and fully supports the standard is critical in that regard.
[00:12:25] Speaker A: Absolutely.
[00:12:26] Speaker B: Do you, do you have much SDI in your plant or is it pretty much an IP based plant?
[00:12:33] Speaker A: It's IP based. We have a, you know, we have 30 odd racks in our technical core, quite a few of which are empty.
It's only in the first two racks that we have a extent of SDI and really that is, that's where our gateways are. Imagine SMPs located past that point. It's, there's, there's occasional piece but very, very small amount.
[00:12:58] Speaker B: Great.
[00:12:59] Speaker A: We, you know, we mentioned budgeting and we mentioned, you know, the constraints that we were up against with this project. And one of the, again, there were a few aha moments as far as the planning and design of this went.
In the early days we had SDI drops all over the building. Because it's a beautiful building. We wanted to do live shots, the capability of doing live shots from all over the facility. And we had, you know, we ran a whole load of copper. We had a whole load of copper planned for that. SDI drops, audio drops.
At some point we said stop, stop, stop. Let's take a step back, take a look. Do we need it?
And really we didn't. We had options for doing live shots. We could use our TPU capability, our Sony live streaming capability through WI Fi if we wanted or if we wanted that, you know, bit of extra assurity drop a strand of fiber, a couple of strands of fiber. We are portable at throw down gateways, we use the Breedel Fusions. Relatively inexpensive, very versatile, very easy to install, take a little bit to configure and set up, but they allow just a very low cost fiber to these various points. With a gateway plug in that gateway and we have that live shot, we cut roughly, I think it was about $400,000 out of our cost by removing that copper which really was never needed. So it is a different mindset, it's getting to think in a different way. And I think that's one of the challenges in a way in the transition from SDI to 2110 is getting a difference. It's a very different thought approach. It's not as process based. There is no block diagram as such as you may have in broadcast. It's a. Everything connects to a single endpoint via the network stack. I absolutely changing that mindset was very important.
[00:14:56] Speaker B: Yeah, and I think that's one of those areas where, and you feel free to expand on this a little bit, but where looking at the total cost of ownership, you know, putting in that, that, that coax that the cost of that integration, that cost of management and the limitations that might give you at the, at the front end versus having aggregation points like the, the beetle fusion boxes or others at your endpoint so that you can change and adapt on the fly to whatever the demand for the day may be. You guys spent some time diving into that with us.
What was that experience like looking at the total cost of ownership approach as opposed to just looking at the equipment part and how you could just quickly get it done?
[00:15:42] Speaker A: Yeah, I mean it's a great question. And the, you know, the answer that we found was pretty surprising. So we had to justify, you know, we again finance budget allocated this project. We wanted to do a comparison to provide justification to Scripps corporate management justifying whichever approach we went, we had a, we took a look at a cost of the equivalent plan, the estimated cost of an equivalent plan in SDI versus 2110.
And we're very surprised that the additional cost to move to 2110, it was marginal, it was almost negligible. Difference being we traded with an SDI based system. You are investing large amounts of money in cabling in power in that infrastructure, the infrastructure to carry the cable, a lot of copper and maybe even more so a lot of cost and services determinations for BNCs, et cetera, et cetera.
With 2110, the vast majority of the cost, a far higher percentage of the cost is with the systems. It's with the network stack it's with is with the gateways, it's with the management systems as opposed to just our pure compass. So we felt an investment in technology to be far better than an investment in, you know, a cable tray full of coaxial cable.
[00:17:11] Speaker B: You guys are using a spine leaf architecture. Was there some impact on the size of your leaves versus your spine that you discovered as the system integration, as the system design and integration I should say kind of came together. And were you kind of surprised by that?
[00:17:26] Speaker A: We were very closely. I mean you know, it would have been nice to have taken a different approach but the, the spy leak worked for us. We made sure with the architect and the design, working with yourselves, working with Arista, that we had a fully non blocking network set up.
You know, monolithic stack would have been great but we, we've not been impacted at all in terms of limited bit rates, in terms of limited bandwidth.
The sizing and the scoping of the SAC is worked perfectly for us.
[00:18:00] Speaker B: And I guess what I was kind of probing for there Andy is is did you find that you needed as much leaf as you as was originally designed or did you find more stuff aggregated and making it back to the spine directly?
[00:18:13] Speaker A: Oh, we, we need it. We did need again one of the complexities we have with this project. And when we started this was we were, we had nine months to 12 month lead times with switches. So we asked to estimate, we had a guess. We had to take a best shot at what the, what the requirements would be.
And as you say that was based on probably in part the same thought process with sdi, you know, a spigot per feed, et cetera with the capacity per port. We find ourselves with quite a bit of excess number of excess switches at the end of the day which you know, portion we'd kind of built in agreements with our agreements with the vendors to get some return and to right size. The, this, this, this, the stack 100, you know, 100 gig connector, that fiber that carries a lot of, a lot of video even at 1080p at 3G 1080p, that's a lot of video.
[00:19:13] Speaker B: Yeah, and that's actually what I was probing for is with the amount of aggregation that is occurring on devices like imagine SNP and other people certainly have those same kind of capabilities, those direct links back to the spine save you quite a bit on the leaf architecture that you might need to have to support smaller, simpler operating devices. And so the leaf architecture shrinks and you're able to take advantage of Those high density lines getting back into the core.
[00:19:49] Speaker A: Yeah, absolutely not. For every piece of equipment for a TVU that's a single feed or nowadays four feeds. For a camera, similar, but for high density systems such as you mentioned, the S and P or, or for our Sony MLS switcher production switches, et cetera, it minimizes the connectivity.
[00:20:14] Speaker B: Yeah. And I think you've found that between the Retle fusion boxes and your Sony TX 30s and that type of approach, we're seeing an increase in devices that do that aggregation and dissemination for you. So that even the fiber strand count is, is, is shrinking because we're able to put so much more data in band back to the spine, correct?
[00:20:39] Speaker A: That's absolutely correct. Yep. Right.
[00:20:42] Speaker B: In terms of future products, you know, we've, we've talked about some of the new things that are coming out. Lots of server based stuff going forward.
What, what are your thoughts there based on the, the, the incorporation of 2110 into your, your, your fundamental architecture in looking at these new technologies going forward, are you confident, are you still a little worried? Where do you stand on that?
[00:21:05] Speaker A: No, I think we're good. I mean I think, you know, there's a, there's a confidence in there. I think you're referring to, you know, microservices and such like we built in with a design. The, you know, some of the, some of the products are always there. Some of the products have that capacity to add to in the future.
The capability of pooling resources was very big for us. The ability to share across different areas, different systems, et cetera.
You know, I mentioned a few MLS switcher. We love the architecture of the MLS switcher in that it's a node. You can have multiple nodes and as you add those nodes they can be shared across different production control rooms. Same with larva. You know, we have one core larval audio board, but the. That is very easily pooled between two, three, four production control rooms. Riedel had taken a very similar approach.
It is, you know, it is the direction. I mean, let me ask you Steve, with the new products that have emerged, you know, since last year, since we started planning, et cetera, I would you say these advancements have influenced the other build outs that you've had in newsrooms, sports stadiums, TV studios. What some of the new workflows or capabilities are possible from that?
[00:22:24] Speaker B: Yeah, I think that it's centered around these kubernetes, containerized microservice architected solutions like what you're experiencing with the Lava Home app for audio. And of course that's expanded well into video and all the major manufacturers are moving that direction. And the cool part about that is that infrastructures that are already set up for an IT centric operation, it's really easy to add these capabilities now to those facilities and it's not that difficult to add them to existing SDI facilities. It's just the cost is so much lower being able to implement those onto a well architected spine architecture. On the networking, the critical thing is that as these devices mature, we're going to see a reduction in the amount of physical hardware, you know, purpose built instantiated solutions and more to things that are running on high speed servers with high rate GPUs. The rate at which GPUs are increasing in their capabilities and up this literally the, the number of gates that are available on them is, it's just, it's unbelievable.
We've seen probably a, a fourfold, well not four times ten to the fourth is what I'm really looking for. We've gone from say 50 million gates or teraflops on, on GPU just you know, six to 10 years ago to now over 50,000 thousand teraflops on a GPU. The ability to just mass process so much information and do those vector translations, it's just, it's incredibly powerful and it's only increasing. That gives us the ability to do more processing on a single server or to be able to spread information between servers to, to optimize how, how they operate. So it's becoming a more IT centric type of operation. And we're finding that coming down even into the mid tier to almost the lower tier range of products. You take a look at a Panasonic Kairos for example, and you can see how much capability that it has in a box that can easily handle SDI 2110 NDI streaming formats, ET cetera. The ability to put that into a plant today is an SDI plant is a little bit daunting, a little bit challenging. But in a, in a 2110 based plant it's BCK. So your ability to, to utilize these new technologies, these new techniques and their AI implementations is just going to be phenomenal. And that's probably the thing that I, I'm seeing that is, is the most powerful is is the deployments of, of AI in creating new workflows or giving you new capabilities within the workflow. It's really starting to come to the fore. And having the processing capability to run that AI is of course really critical. Things like what EBS is showing now with their next generation of replay, where they can interpolate between frames, what got lost as the camera image was moving from one frame to the next and put that information, it's not information that's actually there, but they can put it back in, put it in, substitute it and it's transparent.
It's impossible to see that it wasn't the right kind of information.
Those kinds of capabilities to solve problems are really powerful in improving the functionality of the facility and giving you that next generation capabilities. The same thing is going to be able to be applied to colorimetry, to lens issues, your ability to use a wide variety of luts if you will on image capture, image processing, et cetera, so that you can maximize the quality of the video that you're putting out there, you could certainly do a bunch of that moving the data around with sdi. The real advantage of the IT is the fact that you can now move more data around and within that 2110 fabric and it can be in band with 2110 and it can be out of band. It gives you that IT centric capability to link things up. So I think it's pretty powerful and it's going to only make the operation more powerful in more of a simple way of integrating it than we've seen in the past.
[00:27:14] Speaker A: That's very cool. You mentioned evs. Are there were any other specific product releases that really stand out for you in simplifying those workflows?
[00:27:23] Speaker B: Well, we're seeing IT in the newsroom. The newsroom computer systems are becoming more intelligent and then relying more on these AI capable devices. So we're going to see those move more to servers and much less purpose built gear. I mentioned evs Evers has same kinds of things with the flexibility that they can use with their replay devices. The Dreamcatcher is a phenomenal piece and has lots of capabilities now that, that are expanding what, what you can use it for, how you can deploy it, how you can optimize or customize workflows for specific operations. And again all of that is is based on this move to an IT centric architecture. The other thing that's that's interesting I got to mention here is that the speed with which the development can occur in a microservices kubernetes containerized type of arch. It moves from the purpose built stuff which takes 12 to 18, maybe even 24 months to develop a product, to creating solutions that can be developed within weeks or months as a result requesting a change or an adaptation for a customized workflow. Is much easier, much cheaper and much faster. And because they can operate in isolation on the server, you can experiment with it, you can validate that it works. And if it doesn't, it doesn't take the rest of the infrastructure down. In order to do that with a purpose built device, you got to put a new card in or a new device in and either works or it doesn't. And if it doesn't work, you may be taking down lots of other stuff when that, when that device, when that device crashes. Plus you have to do regression testing on that hardware instantiation to make sure that it works with, with previous versions of operating software or protocols, that type of thing. So it's kind of an exciting place to go to the traffic systems. Let's see, I've talked about Replay.
The sheer processing density is increasing. You guys have something called Master Control Light from imagine running on your SMP processors. Capabilities that it has can be expanded and how they can carve up their, their devices is expanding and, and what those devices are going to look like going into the future is going to expand in terms of its video processing capabilities, its audio capabilities, et cetera. So it's, there's a, there's a lot of cool stuff that's coming and you've got an infrastructure now that can easily accept those kinds of capabilities and devices, most of it being server based going forward.
[00:30:15] Speaker A: So yeah, you mentioned the MC Lite and is again, I talked earlier about the change in way of thinking devices and I think that kind of is an exemplary example of that change in thinking. This is a, it's a firmware that sits within the, one of the processing units within, excuse me, our fabric.
There is no hard switcher. There's no inputs, separate inputs, there's no separate outputs. It's just an integral part of the fabric.
So from a cabling perspective, there was no cabling effectively required to get a master control switcher in place.
But at the same time it takes a difference and a shift in thinking and how that frame was architected, ensuring the MCL Light firmware was installed alongside other devices in that same frame, which allows us to do a reboot if necessary, allows us to do restarts, etc. It is, it is a shift. So Steve, you know, you mentioned the increase in commercial off the shelf solutions and you know, the cost considerations against that. Are there any particular solutions, cost solutions that are proven valuable for balancing cost versus performance? What have you come up against?
[00:31:42] Speaker B: Yeah, I think that's a great question.
We all want to go to lower cost devices, lower cost installs, lower operating costs, and the area in technology that's accomplishing that the best is products that are going into data centers. That's where the largest demand is. And it used to be that broadcast was the innovator. Nobody could, when, when HD signals came out, nobody could do what we could do economically over a piece of coax. In the last 20 years, that's absolutely flipped. What is going on in the IT world is phenomenal in terms of its data handling capability, processing capability and sheer infrastructure transport capability. And it's doing all of that at a lower and lower cost because it's being driven by the IT industry, which is 100 times bigger than what the broadcast infrastructure industry is. As a result, if we continue to ride that wave of innovation and utilize their technologies in broadcast, we can achieve these, these, these lower cost kinds of solutions. So that kind of goes back to what we were talking about with the capability of GPUs, CPUs, et cetera on servers. As the capability of servers continues to improve the capability of the embedded GPUs and may even seeing more FPGAs come in on system on a chip type of solutions, we're going to see the cost of processing or ruining go down. And as a result we're going to be able to do more with fewer pieces, but at the same time also increase our capabilities and take on new technologies at a much lower cost. So we're seeing a dramatic improvement in the capabilities of servers. We're seeing a dramatic improvement in the capabilities of integrating processing like what we talked a little bit about with Imagine. And it can go from the small like what we talked about with Panasonic's Kairos, to very large scale like what Grass Valley offers with their AMP solution where the entire operation can be instantiated in a wire diagram workflow that can change on the fly using basically completely off the shelf servers to create your operation. The only hardware that you have to have outside of the COTS and the network infrastructure is the ui, the human interface components like a keyboard, you know, a control panel or a switcher service, those types of things. So lots of, lots of cool stuff going that direction. And as, as you, as you look at budgeting in the future, that's one of the things to consider is, is the lifespan of the servers and what you can do with those servers as you go to replace them. They don't have to go away. You can move them to less critical tasks or use them to, to do production work in a lower criticality area of your production, move them to or repurpose them to expand your capabilities to do more things in general.
That's one of the other things that I think is really important to understand about this move to not just 2110, but to an IT centric production paradigm is that the, the, the way to repurpose gear is to buy the gear that gives you the, the flexibility to upgrade when needed and reuse that gear for other things that maybe you, you couldn't get to cost wise to begin with, but now you can. Does that kind of makes sense where that's going?
[00:35:34] Speaker A: It does, it does. And you know, the, I think it's interesting in terms of the life cycle of IT equipment, comparing that versus broadcast equipment. Something that broadcast has to be aware of and have to be ready of, you know, it tends to have a shorter shelf life if you like. But at the end of the day the replacement for the something that is off the shelf is not going to be in the same realm as replacing the specific broadcast product.
[00:36:06] Speaker B: Yeah, I think that's a very good point. The amortization time for typical purpose built hardware for, for broadcast was 7 to 11 years. And some of that's driven by IRS code here in the US and it's three to five years on the IT part. So that's really kind of where you kind of need to change your mindset is that I'm going to be replacing my servers probably five years, seven years. You might get some extra time out of it, but in order to bring on the new capabilities, it's a server upgrade, you know, a newer server, but you're not going to have to change the rest of the infrastructure. And that's the key part is changing out a server as compared to changing out a purpose built router or a purpose built switcher.
Processing. Yeah, huge, huge change.
[00:36:59] Speaker A: Absolutely, absolutely.
[00:37:01] Speaker B: So, so there's always a debate between going full IP, 12 gig hybrid infrastructures, all that kind of stuff. And I think we've, we've kind of alluded to how to make some of those decisions.
What, what really broke it for you guys to go full ip? And what would you say to people who are looking at that, what are the key things that you say, you would say that they need to pay attention to in making that decision?
[00:37:33] Speaker A: We were very, very fortunate, you know, we are lucky in that we were building a brand new facility from the ground up and as such it made it a relatively easy decision in terms of our goal to go 21:10.
It allows us to build from the core outwards as a 2110 infrastructure. You mentioned, you asked earlier about and which SDI we have the plant. And it's, you know, it's relatively, it's, it's a decent amount, but it's not overwhelming. If you are replacing say a production control room or you're replacing a central router or any such island within the facility, it becomes more of a question. You have to look at, you know, what are you aiming at achieving with that? Are you looking at five, you know, formats that maybe use five, five years down the line for ourselves next gen TV and obviously opportunity for increased resolutions for 1080p streaming, you know, I mean Netflix and those are constantly streaming at 4K. What about broadcasters? Do we want to be doing the same and are we looking to that resolution, the hdr? Where are we with HDR and HDR rollout? Do we anticipate that on a broad basis in the near future?
If the answer to those questions is yes, then I think it's worth looking at the potential of an island and build out as far as a plant goes.
Remembering that gateways, as I say, they're not the price they were, but they're still expensive and the hardware is still, you know, there's a price associated with that. But one of the benefits in what many of the gateway manufacturers are producing now imagine S&P's as I mentioned earlier, that's our use. They have multiple, I think they call them personalities for those, those video processing units. Because it's more than just a gateway. If you install a production control room now, which is an SDI or sorry, a 2110 control room and you have to gateway everything in gateway, everything out.
Two or three years down the line, you decide to replace your core infrastructure, your core rider and move to 2110.
The gateways you purchase for that can be repurposed. You can add different personalities and the COVID switcher, it can be used for color grading, it can be used for so many different purposes. So the acquiring these gateways, although it might appear to just be a short term requirement, they have a long life in them. The personalities give them the scope to be reused and reused in many different aspects is required.
[00:40:24] Speaker B: That's, that's a good point. I think one of the key things that I, that, that I, that I'm aware of is, is you got to look at the cost of, of the, the equipment, look at how much of it do you, you're going to have to, you're going to have to change, look at what your Long term usage is going to be. And then the kind of those, those more intangibles is what is, what is it going to cost me to train my people? What's it going to cost me to, to manage it and operate it? And what are the, what are the service contracts going to look like on these products versus SDI versus, versus 2110?
You guys, you guys looked at a lot of that and, and I think you found that that long term going to the IP based was going to save you some money. Correct?
[00:41:09] Speaker A: It, it doesn't, it doesn't. I mean the, you know, we, you mentioned support agreements and you mentioned total cost of ownership early on.
You know, the support agreement for a, our previous building, we had a, you know, very old rider and it worked and there was no support, you know, agreement needed for that. If we needed a spare, we'd thought the manufacturer could provide one. If not, we might go hunting on ebay. Yes, we've done that many a time.
Whereas with the new architecture really, you know, support agreements become more essential. So I think as you move into whether it's an IP, whether it's 2110 running on IP or whether it's a commercial off the shelf servers running software, the support becomes more essential. So I think you have to accept there is going to be an increase in support and ongoing costs. From that perspective there are other aspects which do bring bring down costs. You know we talked earlier about the footprints again we designed. When we started designing our new facility we had no idea how power compared. We had no idea our rack space compared.
So we used what we had as a basis as a, as a starting point for that. We designed a rack room in which we could have built an SDI facility.
I think when we started designing a rack room, we didn't know if we go on SDI or 2110. Same with power. You know, we, we looked at our power usage at our facility. We had a UPS which give us 5 minute runtime for all of our broadcast infrastructure, allowing the generator to kick in which you know, should be no longer than 30 seconds. So five minutes was, was more than enough.
The rack space. We got a lot of empty racks in our room. Now we got a lot of room for expansion and we didn't install. We had another 6R air racks which we didn't even need to install if we needed, we had the space but we, we took those out because we didn't need it otherwise. When we started commissioning of our systems, we were pretty much blown away. I was blown away When I looked at the UPS and we tapped the panel with the display and a little GUI and it said I had 120 minutes of runtime.
And that gives an indication of how much more efficient these systems are when it comes to power costs and power footprints, et cetera. I don't think it was, you know, it's not as quite a 16 fold improvement in terms of power. I think we, you know, we. There are other systems outside of the broadcast which would have been drowning a bit, but is a significant reduction in power footprint.
[00:44:05] Speaker B: Yeah, I think that's one of the things that is hard to get your head wrapped around is all those times that you do the serialization deserialization on an SDI signal versus having that signal available to anybody, anytime, anywhere in the plant using multicast, it makes a big difference.
You just physically don't need as much power because you're not doing as much processing to get it into a format that might be needed. There's some architectural considerations though. You mentioned that you don't need as many racks. Your engine room could have probably been smaller.
How about other architectural considerations like your cable runs and those kinds of things? Outside of the engine, the technical core.
[00:44:47] Speaker A: Fraction of the mount, we have a lot of space in our cable tray.
Um, we made an early decision not to go with computer flooring. Um, we removed that again. We looked at what was being ran, we looked at how much space was required and it just, it didn't make sense to do so. We didn't need it. It was fairly easy to take cables overhead looking at the cable tray. And our plan is very open. We went with a kind of an industrial architecture look about the facility.
Many of the corridors go up to the roof of the plants. The facility 25 foot from slab to, to roof. And we have cable tray, you know, running along those corridors, high up in those corridors. So it's all exposed, it's all open and it creates a very good luck. Those cable trays are astonishingly empty compared to any other facility that I've been in.
Even into the studios where, you know, you have cameras, we've got monitor walls, you have prompter, you've got bsps, you have so much in a studio. And the amount of cable going into the studios is I would estimate between 10 to 20% of what it would have been in a, in a traditional studio. So, you know, the, from the architectural aspects, it's big change.
[00:46:17] Speaker B: Yeah, that's, that's cool. Did you find that there were some other things that you were able to accomplish, like maybe redundancy that you, you couldn't get to in a traditional SDI plant.
[00:46:26] Speaker A: Yeah, I mean, it's very hard to. If you run an SDI router 256 by 256 rider, which, you know, is a kind of an average news, local news station riders, some have bigger, some have smaller. It's very difficult to have two routers one side by side. And you know, one fails to easily and seamlessly fail over to a second router.
You know, I mentioned earlier there are a lot of substandards that form part of ST2110. One of those is ST2022 7. It was developed for a year earlier advertising SDI video standard, but it carried over into 2110.
We have two core stacks, two core Arista media switch stacks. We have a red and a blue. And 2110, it, it does, or 20227 doesn't just give us redundancy. It, it distributes packets where it feels best. It's dynamically switching. It could be. We could have packets going through the red, going through the blue. But if we drop one of those stacks, it is seamless bailover to the remaining stacks. So we have absolute redundancy with that. If we have a cut or fail to any of our core equipment, it will stay up. It will seamlessly switch over from blue to red or red to blue. So the capability of building in redundant and resilient video paths is a huge benefit for us.
[00:47:56] Speaker B: Great.
Looks like it's about time for us to jump into some audience questions. Is there anything else that you'd like to share before we start taking some questions?
[00:48:04] Speaker A: I think one of. Again, one of the. We talk. I talk about many times about mindsets and you know, I mentioned the 256.256 router, video router, and again, the core infrastructure. Something important to understand the core infrastructure. Within 2110, every single endpoint you have, video endpoint you have, or audio endpoint you have is either a source or a destination. On that 21:10 equivalent of that video router, we went from, you know, 256 squared to I think it's between 2 to 3,000 inputs and 2 to 3,000 outputs of our router or equivalent to the router. Not only that, every single video signal you have is split by definition. And according to the standard, video is spun off, audio is spun off. Ang data is spun off. So you have this incredibly flexible capability of writing anything from anything to anything. And to be writing either all Video, audio, ank or to separate those. You know, you think of closed captioning in the past, closed captioning, you had an encoder to add closed captioning to that ank on the SDI. Now it's a simple route. We just route a dash 40 stream out of our closed caption encoder or a closed caption creator, I should say, onto our output stream. A little bit more complex in real life than the theory, but it works. It's. It's an incredibly versatile architecture, but at the same time it's an incredible, it adds an incredible complexity. So care is needed.
[00:49:51] Speaker B: Yeah, great. I think that kind of ties into our first question from the audience. Brett has a question of how many channels of audio did you allocate and what complications did that add to the IP matrix? It's a great question.
[00:50:04] Speaker A: It is. I mean it's all, again, it's all in GUIs, it's all in software. There's no patching, there's no hardware, there's no cabling. It's part of it, part of a stream. So and to be honest with you, the creation of ifbs, it, it increases in difficult. You have to have people who know in depth, both the comm side as well as the, the dash 30 larvo side. For us, I mean I'm talking, you know, with us, it's real for our comms, it's larva for our audio board and the creation of those IPs, it's a lot more difficult.
But once you get your again, the mindset behind it, again it increases the capabilities, the flexibilities of what you can do within those systems so that the.
[00:50:55] Speaker B: Setup and commissioning is a little, is much more complicated. But the scalability and applicability is the flexibility. You're treating flexibility for complexity or complexity for some flex, the added flexibility.
[00:51:10] Speaker A: Absolutely. And I think that's, you know, that's true across the entire platform.
[00:51:16] Speaker B: Great. Another question Here is does Denver7 have a plan for ENG or EFP contribution from multiple locations with a centralized post repository?
[00:51:26] Speaker A: It's a great question. It's something, you know, I don't think that changes too much with the transition from SDI 2 to 21 times storage, file based movement of files, etc.
It's pretty much the same as our, as our facility we use Scripps is created scripts. We have some very, very smart people in this company. They've created in AWS systems, Amazon web systems, we're pros for a video and mechanisms that we can Share. Every single station can dump into this AWS storage and every single station can pull from. So from a 2110 viewpoint, you know, not too much has changed with that. Say, with eng, we use TVU extensively and TVU for a live shot.
Whether we're sending from the field to our stells or to the our station or whether it's a national story and many script stations want to pull down from this feed, it's pretty much the same.
2110 does have capability, you know, remembering and video signaling. 2110 it is. It's a lot of bits in there. There's a lot of bits. But if you're running 1080p as we are, you're talking 3 gig and 3 gig. Trying to distribute orbit, any type of one is very, very difficult, if not impossible.
There are substandards which are out there. There's 2110, 22 which can use JPEG XS to distribute video so natively within the 2110 formats across at a much more efficient bit rate. I don't know how well that's been used. I'm not sure, you know, if it has been adopted to any extent, but it's out there, it's available.
[00:53:18] Speaker B: Yeah, and I think that's a really good point that with 2110 there is no defined limit as to what kind of data rates you can do. I was doing some investigation for another client at one point here recently and they were looking at doing 4K 8K at 4, 4, 4 and the standard supports it. The only question is finding manufacturers who have implemented interfaces that can accept that kind of data rate and process it. So it's a standard that is capable of doing a lot, even if it's not what everybody's going to do. I got another one here that I want to touch on because this was kind of something that we discovered during the integration and had to kind of make some adjustments as things went along. And that was vendors that maybe didn't fully incorporate 2110 into their standard or their implementation was, shall we say, not as cost efficient as it should have been. And how did you handle some of those kinds of things?
[00:54:30] Speaker A: Yeah, I mean, on the whole, we find interoperability to be excellent. Vast majority of manufacturers work very well right from the beginning.
We do subscripts. We do have some specific systems we do use have some specific hardware across many different stations.
Some of the systems use, you know, lower cost. We've been looking for how we can do more for less in a production capability. We're looking for Layers of production capability. And part of the planning for that did use some Blackmagic components. Blackmagic did launch a range of 2110 a year a couple of years ago. And we planned around the implementation of some of those products.
It kind of caught us out. The SD2110 was ratified, the NMOS was not. And we find it very difficult to integrate with those products how to take a fairly fast turnaround at the last minute and use a traditional decklink card with a gateway. So we, you know, if it doesn't, and again if it doesn't work in 2110, often there is the option of the SDI. You can take the SDI and can get can Gateway. So it's not a deal breaker. You just have to think around that.
[00:55:51] Speaker B: Yeah, I think that's, that that's kind of the point I was, I was driving at is one of the key things you can do is look at the cost of an SDI based solution for a given product and then look at what the cost increment Is for their 2110 solution and compare that to the gateway cost and sometimes you'll find that their implementation is 2x what the gateway cost is and you make the appropriate decision.
[00:56:18] Speaker A: Absolutely. I mean I will name the vendor, but we did find one particular product which we use quite extensively and it was available with Gateway. We could we or with the 2110 inbuilt capability. But the cost of doing that, I think it was two and a half, $3,000 to add that capability instead of a, you know, a 500, 700 foot gateway.
[00:56:41] Speaker B: Yeah, yeah, excellent. We're kind of running out of time here, but I want to get one more question in here that I think is kind of important.
The question is how do you line document the devices and workflows?
[00:56:55] Speaker A: That is a really, really good question. And as I mentioned earlier, the reason the workflow is not there, the block schematic of a camera into a distribution amplifier into a router into a switcher, et cetera, et cetera. It doesn't exist. It really doesn't exist.
There are different products out there which simulate and can give a, you know, a virtual, virtual block diagram that can give imagery of that. We're not really using any of that. We don't have any that is fully ratified. We're hoping maybe down the line we will and that was a big concern of us in the early stages. How do we troubleshoot from A to B to C to D And really that is again, it's a change in approach. We learned, our staff have learned really to look through the rider tables. We can see every source, we can see every destination, we see every level and kind of point through those tables to troubleshoot a path. And we've had to many times, you know, we've, we went live as mentioned back in July.
It's not been without, you know, some initial issues, it's not been without some recent issues. But every single week that's gone by it's got smoother. We've, we've, we've got more efficient, way proficient and those issues have, you know, dwindled to relatively smooth. Smooth running now and learning how to troubleshoot in this environment has been essential to allow us to do that. I would love to see the products. I know that there has been many discussed and you know, we, we've seen many demoed at the trade shows.
That might be the next step.
[00:58:41] Speaker B: Yep. And I think that's a, that's a cost benefit analysis that you have to take into consideration.
[00:58:46] Speaker A: Correct.
[00:58:46] Speaker B: So. Well, unfortunately, that's all the time we have today. Big thanks to Andy for joining us and his willingness to discuss his implementation of 2110 at Denver7. It was a fascinating discussion. If you're exploring newsroom or facility wide upgrades, please feel free to reach out to us here at Key Code, any member of our integration team. Our team is standing by to help you figure out the best path forward. This podcast will be available on Apple, Spotify, YouTube and other social media channels. And thank you again for watching.
[00:59:19] Speaker A: Thank you Steve.
Thanks for watching.
[00:59:22] Speaker B: Broadcast to post, Please make sure to.
[00:59:24] Speaker A: Subscribe to the podcast to receive Future episodes, follow keycodemedia on LinkedIn, Twitter, Facebook or Instagram to receive news on additional AV broadcast and post production technology content. See you next time, folks.
