CBC Releases Tens of Thousand of Records

Today the CBC posted tens of thousands of documents released under the Access to Information Act since 2007 to its corporate website.

The index of all the documents is available here.

Posting the documents online makes them available to the general public, not just the person who made the request. The CBC also committed to updating the site as new documents are released.

CBC President and CEO, Hubert Lacroix said of the announcement ”we’re taking yet another step in our efforts to give Canadians the information they need to understand the way we manage their money.”

7 Comments » Email This Post
  Under the Hood Posted at 8:09 pm (25 Nov 2010)



New Weather Section!

Earlier this week CBC.ca launched a brand new weather section. The new weather section has been re-written from the ground up using some pretty cool technology to get the forecast to our users as quickly as possible.

Automatic Location Detection
Using technology from our Content Delivery Network, Akamai, we’re able to figure out with 95% accuracy where in Canada you are visiting the website. We take this information and automatically redirect you to the closest weather station.

This technology isn’t perfect, so you might not get your preferred location automatically. If that happens, just type in a location you would like to the the weather for and click on the “set as my default” link to save your preferences.

Easier To Read Forecasts
The weather pages now include larger icons and text to better understand the forecasted weather. In addition, we have included radar and temperature maps from the CBC Weather Centre on every page. We have also included a handy way to change between Celsius and Fahrenheit temperatures!

Improved Weather Warnings
The weather warnings now link directly to Environment Canada. As well, on the front of every regional page, you will get a list of warnings that affect the region.

How It All Works
CBC gets weather data from a variety of sources including: Environment Canada, WSI, and our own Weather Centre. These various feeds are delivered to CBC.ca where a Java application called “weatherconditions” parses the data and outputs it as flat HTML to the website.

The application uses two libraries to do the heavy lifting. One called “smooks” to parse and process the XML and CSV data that we get from the data sources. The other is called “freemarker“, which is a templating language to place the various pieces of information on the HTML pages.

The weatherconditions application is constantly running on our server and processes the data sources every 15 minutes.

1 Comment » Email This Post
  CBC.ca web site, Under the Hood Posted at 1:13 pm (21 Aug 2009)



A few changes to the site…

There have been some subtle changes to the site that havn’t really been advertised. I’ve taken a moment to outline them for you:

Mobile Site Revamp
There were two major changes to our mobile website. The first was the outsourcing of the site to a third party called Quattro Wireless. They allow us to automatically reformat the mobile site depending on which device you are using. So the site you see on your Blackberry will be formatted for the Blackberry. While a visitor from a cell phone will have the cell phone version of the site displayed to them properly.

The second change we made was to automatically detect if users are coming from a mobile device if they visit www.cbc.ca. If they are, then they will automatically be directed to our mobile site: m.cbc.ca. There were a few bugs with this auto-redirection, but they have been ironed out this week.

This way, you have one url you need to remember: cbc.ca. If you visit from a phone, blackberry, or desktop, you will automatically be sent to either the mobile or regular website.

Flash Video
We have had flash video on the site for a while now. We’ve recently rolled this out to the news areas of the site, specifically The National.

This new video format allows us to provide higher quality video at a lower bit rate. It also allows us to almost guarantee that the video will be played on all platforms as the flash format is supported in most browsers.

You can expect to see Flash video to slowly replace all the windows and real media video on the site. We are currently testing live streaming with the flash player as well.

Anything Else?
There are some pretty cool projects going on that I will reveal over the course of this year as they roll out. So stay tuned!

28 Comments » Email This Post
  , CBC.ca web site, Under the Hood Posted at 1:31 pm (06 Feb 2009)



Moving On Up!

CBC.ca has moved from the second floor up to the ninth floor over the course of the past two weeks. Us, along with the finance department were the first “guinea pigs” for the office space redesign.

Because this is going to be our permanent home, we were literally given keys to the blueprint. After playing around with little chairs, tables, and desks on a floor plan the workers starting building our “dream office”. In 7 months or so, they were complete!

You can see a few select photos of our new space from my co-worker Julian Dunn on his flickr page. Additionally what our old space looked like is also available on flickr.

3169021049_6e586a5fb1

13 Comments » Email This Post
  , CBC.ca web site, Under the Hood Posted at 12:18 pm (22 Jan 2009)



Scheduled Maintenance

Early Monday morning (01:00-05:00 ET Monday January 12, 2009) we will be taking the site off-line to perform some much needed maintenance on our storage device.

The CBC.ca website uses a lot of disk space. All of the media, news stories, podcasts, etc.. add up. We store all of this information on device produced by a company called BlueArc. All of the web and application servers “mount” the BlueArc to serve the website content. There is approx. 3 terabytes of data that is used for the website. 3 Terabytes is a lot of data, to put it in perspective:

1 Terabyte would be equal to 50,000 pages of printed material.
1 Terabyte of seconds would be 32,000 years.
850 Terabytes of storage is used for all of the pages in the Google search index
2,250,000,000 Terabytes of data in 1 gram of DNA

What will I see during the outage?
Because we need to take the BlueArc offline to perform the maintenance, all of the pages in CBC.ca will not be available. However, we are going to be providing an extremely limited site (what we’re referring internally as the “skinny site”). This site will be extremely light-weight and only offer the latest news.

A preivew of the site during the maintenance.

A preview of the site during the maintenance.

All other content will be unavailable until the maintenance is complete.

If you have any specific questions on how the BlueArc works, or why we’re doing this maintenance, please feel free to post a comment.

3 Comments » Email This Post
  CBC.ca web site, News & Journalism, Under the Hood Posted at 3:42 pm (08 Jan 2009)



The 2008 Federal Election: Part 2

In part two of two of my “Federal Election” series we are going to talk about two items. The first, which is the brain child of David Raso, is the postal code/riding look up. For that, I’m going to hand the keyboard over to him:

My Ridings
Hi, I’m David Raso a Senior Architect of Front-end Development at CBC.ca and Blake asked me if I could explain how one of our Election modules work. We wanted to build a module which would show you information and results for the riding you live in, so the My Riding module was created. This module allows you to enter your postal code and then displays information about your riding – it’s on every page of the Canada Votes site.

riding

No Database Needed!
Sounds simple enough right? The simplest solution would have been to hook up the form to some sort or dynamic backed code written in Java or PHP which would search though the postal code database we got from Statistics Canada and display your riding info. Well because we get so much traffic on election night we have to build everything to be extremely stable and scalable. So using a dynamic application was out of the question – so then how do you provide a postal code look up database without a database? We could use some cloud computing (like Amazon or Google) or big complicated Akamai cached edge applications – but we like to keep things simple.

Everyone in meetings would say: “It can’t be done”, then you sit and stare at the white board. Few minutes later we came up with one of those “It’s so crazy it just might work” ideas.

When you enter your postal code into the form and hit search – we don’t connect to a database and find that postal code – we change your postal code into a url and using AJAX (a JavaScript programing technique) we fetch a flat HTML file which contains the ID of your riding. We then use that ID and AJAX, again, to fetch a flat HTML of your riding info. This is all done by your browser so all our web servers have to do is return those tiny HTML files your and computer does all the heavy lifting.

How It Works
So if you enter M9A2X1 we change that into “http://www.cbc.ca/news/canadavotes/myriding/postalcodes/m/m9a/2×1.html” we then download that file using AJAX. The files contents are stored as JSON (JavaScript Object Notation) which the My Riding JavaScript can easily understand and obtain the ID for your riding. Once we have your riding ID we then use AJAX again and download the file for your riding “http://www.cbc.ca/news/canadavotes/myriding/ridings/130.html” then all we have to do it display that HTML file in the My Riding module. Easy Peasy, Lemon Squeezie.

That’s all good and it works but you might be asking your self: “Does that mean you have a HTML file for each and every postal code in Canada?” – yup you’re correct. Over eight hundred thousand – 827,018 files to be exact. We took the database from Statistics Canada and created a script that would turn each postal code into a flat HTML file that contains the name and ID for the ridings in that postal code – some postal codes have multiple ridings, try out K0A1W0. We then uploaded the 3GB folder to our webservers – and because they don’t change we can cache all those files using Akamai.

On election night when the hundreds of thousands of users try to find their riding there is no database to go down, or application server to be bogged down. Just one small cached HTML file to download!

Comments and the Black Out
This is the first Election where commenting was allowed on CBC.ca. One of the issues we faced were users posting election related results in their comments.

This was an easy solution. We asked the 3rd party company that is responsible for moderating our comments (called ICUC) to not approve any comments for publication until the blackout period ended.

Add Comment » Email This Post
  CBC.ca web site, Under the Hood Posted at 3:50 pm (05 Dec 2008)



The 2008 Federal Election: Part 1

The latest federal elections introduced some new tools for site visitors to use on our site.

You were able to use your postal code to get riding information, view real time results on an interactive map, view live video, and talk politics by posting comments.

I will be talking about these applications in detail in the next few posts.

Blackout
Providing these features proved to be a challenge. The Elections Act doesn’t allow us to broadcast results to areas of the country where the polls haven’t yet closed. Therefore we were not publishing any results on the website until 10PM ET. This is known as the “blackout” period.

Flash Crowd
Because of the blackout wasn’t over until 10pm ET, that’s when the majority of users came to the site. Let’s take a look at the traffic graph for that particular evening.

electionsgraph1

The date/time in the graph is in Eastern Time. To give you an idea of the scale of the graph the top of the Y axis would be a value 3.8 times higher than at the bottom. The red arrow indicates the peak, which lasted for approx 10 minutes. As the evening progressed the traffic subsided.

“Elections Mode”
Because of the blackout we had to figure out a way to display results on the site the instant 10PM rolls around. Here’s how we did it:

All of the site assets were uploaded to the webservers hours in advance. We were able to control the display of these assets using Server Side Includes (SSIs).

SSIs allows us to hide or display certain html based on the existence of a variable by using if statements. Here’s an example:

<!–#if expr=”${electionsmode} = on”–>
<results related HTML goes here>
<!–#endif–>

We include a file called “vars.html” at the top of every page which defines the “electionsmode” variable. When it’s time to go live, we just change the “electionsmode” variable in vars.html from “off” to “on” and the new content appears instantly on the site.

Getting The Data
The elections results are delivered to us from another IT department. It is the exact same data feed that is used on-air. We process the results every 30 seconds and spit out XML data for our applications to use. The process works like this:

  1. IT FTPs the raw election data results to our server.
  2. CBC.ca’s Java application sucks in the raw data.
  3. The application applies business rules, calculates percentages, then outputs XML files.
  4. Process is repeated again in 30 seconds.

The XML files are used for the flash map and riding result pages. Because this process only happens every 30 seconds, you might see a delay from what you see on Television. This is because TV does not have this post-processing and transforming that we do.

The Flash Map
The flash map was developed by Tom Klepl, a very talented flash developer. He worked closely with the system administrators to ensure that the map was designed in such a way that it played well with our CDN (Akamai).

The map allows users to zoom in to their riding and see real time results of who is winning the race. As well, it gives a general overview of all the ridings in Canada. One of our interface engineers, David Raso, produced the following time lapse video of how the country voted on election night.

In Part Two
In my next post, we’ll go into further detail on how the postal code search works (hint: no php, java, or database required!) as well as how we dealt with users who would post election results in comments!

2 Comments » Email This Post
  , CBC.ca web site, Under the Hood Posted at 4:09 pm (20 Nov 2008)



Under the Hood: Going for Gold

Sadly this is the last Olympics that CBC will be covering for a while. That said, this is also the most well covered Olympics in CBC history. Partly thanks to the Internet and cbc.ca.

Don’t Feed The Animals
cbc.ca/olympics provides you with up to 12 online “channels” where you can watch Olympic events live. This is possible thanks to the folks in Television (specifically Web Presentation group), encoding software by Digital Rapids called StreamZ, and Akamai.

The Olympic feeds are back hauled by Television to Toronto where they are encoded by the Digital Rapids boxes. They are encoded in Windows Media v8 at 500kbps. This stream is then sent to Akamai for distribution.

In most cases the video you see are raw feeds from the venues. This means that there is no commentary. The only audio you here is ambient sounds from the event.

A lot of people like these feeds as they provide coverage for their sport even if it is not available on television.

Canada Only, Please
Due to licensing restrictions by the IOC, only Canadians are able to view Olympic coverage provided by the CBC. As such, we’ve had to use technology from Akamai to ensure that we follow the rules.

The Akamai method of “geofencing” (as we call it) is more sophisticated than what I described earlier. Multiple methods are used, some of them are:

  • Your IP Address. Using a database of known IP blocks and locations.
  • Which Akamai DNS server you use. When you look up a host name (like www.cbc.ca) you hit a specific Akamai DNS server. Akamai knows which DNS server is in which country and uses that information to figure out what country you’re in.
    BGP Metrics. In a nutshell, Akamai looks at what other servers/routers you go through to get to the stream. If those are in the country Akamai thinks you’re in, then the confidence level goes up.
  • Timezones. Using a piece of javascript on the client side. We figure out what timezone your clock is set to. If it matches with one of the timezones in the country Akamai thinks you’re in, confidence level goes up.
  • We have people who watch you. ;-)

The methods listed above, plus a bunch of others combine to give a Country accuracy level of 99%.

Trends
Because of the twelve hour difference (in the Eastern Time Zone) quite a few of the events happen late at night and early the next morning.

The above graph shows the number of people watching the streams over the past seven days. The majority of the watching happens between 8pm and 12am EDT, and 6am and 12pm EDT.

Some days are more popular than others (like August 14, versus August 16). The little red arrow you see on August 19, at around 12pm EDT is when the most users were watching the streams ever.

I suspect that the closing ceremonies will generate the most streaming traffic during the Olympics.

12 Comments » Email This Post
  CBC.ca web site, Specials, Under the Hood Posted at 2:11 pm (20 Aug 2008)



New CBC Programming guide launched

Under the Hood column

This past week we released a new version of the CBC Program Guide. This was a much needed upgrade as the old version was virtually useless because it constantly crashed.

The Nitty Gritty
This new version is written in Java using Struts and Hibernate. These are enterprise frameworks that eliminate the need to create custom libraries. These “custom libraries” are what caused a lot of the instability with the previous version of the Program Guide.

The previous version stored all of its data in an Oracle database, with no expiry policy. This means that you could go back years to see what was on CBC Television in 2003, for example. The new version stores all of its data in a PostgreSQL database with an expiry policy. There really is no need to know what was on CBC.

The new guide is also extremely extensible. Able to output in multiple formats including HTML, XML, and JSON. This allows other CBC developers to leverage the data located in the guide.

Quick Rundown
Program Guide information is available for all CBC properties: Television, Newsworld, Radio One, and Radio 2. It also includes A to Z guides of all CBC Programs and Personalities. The new guide has been redesigned so that it is easy for you to view what is currently on air right now, which is highlighted in blue.

You are able to “segment” your day into early morning, morning, afternoon, or evening. So you only can only see 5-8 hours blocks at a time. You are also able to view the full day, or a schedule for the entire week. Clicking on a show title will bring you to the program page which will allow you to see air times, plus a description about the show and its personalities. You can also filter your schedule by program category. So you can only see Sports, Comedy, Drama, etc.

The print friendly version of the guide is well formatted and easy to read.

The Future
With the new Program Guide framework we are able to provide a lot of new features. Some ideas floating around include RSS feeds of your favorite show (air times, descriptions, etc..). The ability to include program/personality information in our search engine is also a possibility.

The new Program Guide will be used during the Olympics to allow you to know exactly what event will be on-air when. You will be able to access this information from the Olympics page or the Program Guide main page.

Expect to see more features and pages that utilize the new Program Guide in the near future!

6 Comments » Email This Post
  Under the Hood Posted at 2:34 pm (22 Jul 2008)



Farewell Webtrends, Hello Hitbox!

And… farewell Blake.

The following is the last of the “Under the Hood” columns that have appeared on this blog for more than a year, courtesy of CBC.ca tech guru Blake Crosby.

Blake worked with CBC.ca for almost six years, first joining the team to babysit the servers during the Salt Lake City Olympics. He went on to work on other Olympic and elections sites, among others, and won an award for his work on CBC.ca’s Media Resource Locator tool (see his earlier column.)

Blake can fly!Blake has moved on to a company called VerticalScope. Long term, he’s working toward a career in aviation – you can track his progress on his flying blog.

Thanks, Blake!
~PG
————–

Farewell Webtrends, Hello Hitbox!

There have been some behind the scenes changes in the way we process and crunch the web server log files.

Webtrends
The software we were using previously was called Webtrends. It processes the raw log files from the web servers and produces graphs and charts.

The main advantage to using Webtrends is the fact that it processes the raw web server logs. Anytime someone fetches content from our web servers, it is recorded in a log file. Whether this be a mobile phone, Internet Explorer, your grandma’s 386, or your text only browser – it’s all tracked.

Items such as your IP address, the page you were requesting, the type of browser you were using, and the date and time were recorded. This provided a solid source of data to process.

One of the downsides was the Webtrends limitation that the log files needed to be in chronological order. This is impossible with our website, as we have many different log file sources that are all out of order. There was a lot of overhead to merge all these log files into a chronologically correct source of data for Webtrends.

Changing Business Requirements
HBX Analytics With the recent “upgrade” of the internet to Web 2.0, CBC needed to upgrade their website with more “Web 2.0″ features. This included items such as the most viewed stories, or most e-mailed stories. This real time data was available from the web server logs, but Webtrends couldn’t process the data fast enough for it to be useful.

This is where Hitbox, our new system, shines. The Hitbox product comes from a company called Visual Sciences (formerly WebSideStory.) It works the same way as Webtrends, except it offers real time data of people visiting the website. This is not done using log files, but javascript instead.

For every single page you visit on CBC.ca, a cookie for “a.cbc.ca” will be set. This cookie is used by Hitbox to track your movements throughout the website, and is recorded in real time. Although no identifiable information is recorded, we can see how individual users use the website.

That means content producers can track the performance of various areas of their sites in real time – understanding what stories are most popular, the times of day with heaviest usage, the most common navigation paths through the site, what links users follow to and from stories, and so on. By watching specific live stats instead of waiting for a report, the content itself can better reflect users’ actual behaviour.

7 Comments » Email This Post
  Under the Hood Posted at 11:16 am (28 Nov 2007)



Wiki Mania!

Under the Hood

It seems that computer programmers like to use Hawaiian names to describe their applications. We use two such applications:

  1. Media Wiki (“quick” in Hawaiian is “wikiwiki”)
  2. Akamai (Hawaiian for “intelligent” or “smart”)

CBC has their own wiki and uses the same software that powers The Wikipedia called Media Wiki. Anyone and any department is welcome to use it. We currently have 259 articles and growing.

CBC Wiki

CBC Wiki front page. Click for larger

Some of its uses:

  • Training guide for new hires
  • Personal pages for employees who use them as note pads or a place to keep track of project progress
  • Information about infrastructure, policy, phone numbers, etc..
  • A listing of standards to be used by content producers

The wiki is only available to CBC employees at wiki.tor.cbc.ca . It is pretty technical as only the IT departments are currently using it but should be useful to other departments as well including Radio and TV.

1 Comment » Email This Post
  CBC.ca web site, Under the Hood Posted at 2:51 pm (07 Aug 2007)



What’s The Password?

Under the Hood

Manging multiple machines or accounts involves remembering quite a few usernames and passwords. Every member on the team also has to use the same usernames and passwords as well. The challenge is to find a secure way of storing these login credentials so that everyone has an easy way to access them.

Originally we stored all the usernames and passwords in what we called the “Password Book”. This book was a simple notepad with the pertinent information stored in a cabinet close by. The main problem with this system was that the information was not available unless you were actually in the office. This made on-the-road troubleshooting almost impossible.

Currently we store all of the information in a blowfish encrypted database. The database is stored on a Wiki page (which I will talk about in another post) where it can be downloaded and stored on a usb key.

This database is then read by a program called Password Gorilla. The advantage to using Password Gorillia is that the client is available on a multitude of platforms, including mobile ones like Windows CE (but not Black Berry).

In reality, we just need to remember one password: the password to the encrypted database, and we have access to all the usernames and passwords we need in order to do our jobs.

10 Comments » Email This Post
  , CBC.ca web site, Under the Hood Posted at 9:54 am (10 Jul 2007)



We Know Where You Live…

Under the Hood

… 82% of the time.

Weather Widget

One of the cooler features of the new site design is the weather “widget” in the top left hand corner of the page. Some people might have noticed that the weather report being displayed is for the major city that is closest to you. How did our website know what city you are in?

It’s All About The IP Address
Every computer on the Internet has an IP address. These set of numbers allow networks to send the data you requested to your computer and not the one down the street. The IANA is the organization that keeps track of IP addresses on the Internet. They assign a block of addresses to 5 different portions of the world: Africa, Asia/Pacific, North America, Latin America and the Caribbean, and Europe, Middle East, and China. The fact that this division is made is the first step in figuring out your physical location based on your IP address. At this point, we can narrow down which region of the world you are located. Obviously not accurate enough to use in our application.

In each one of these regions there is a organization that then redistributes IP blocks allocated to them by the IANA to various people or companies in that region. For our discussion, we will focus on ARIN (American Registry for Internet Numbers) which handles IP addresses in North America. ARIN would be similar to a telephone company that distributes telephone numbers when you move to a new house. Except in this case, they hand out IP addresses to ISPs and Corporations.

It’s Public Information
ARIN (as do all “registries” as they are called) require that “owners” of IP addresses have a physical address on record. This information is available to the public at various locations.

Lets take a real world example: My IP address for my computer at home is: 24.137.199.17. If I type that address in the ARIN website, I get the following result:

Aurora Cable Internet ACI2 (NET-24-137-192-0-1)
                                  24.137.192.0 - 24.137.223.255
Aurora Cable Internet HE-24-137-192-0-19 (NET-24-137-192-0-2)
                                  24.137.192.0 - 24.137.223.255

Great! I know that a company called “Aurora Cable Internet” is responsible for my IP address and any other IP address that is inside the ranges listed above.

The string after the company name (“ACI2″ and “HE-24-137-192-0-19″) will be able to provide us with more information about that company. Plugging in “ACI2″ into the same search box on the ARIN site reveals (among other information):

OrgName:    Aurora Cable Internet
OrgID:      ACI-38
Address:    350 Industrial Parkway South
City:       Aurora
StateProv:  ON
PostalCode: L4G-3L6
Country:    CA

Now we all know that I live in Aurora, ON! More importantly though, we have just associated a physical location with a IP address.

What’s this 82% business?
The database that we use is only 82% accurate within 40kms of true location.

So what happens if you are in the 18%? The weather defaults to Toronto (further proof that we’re the centre of the universe). From there you can click on the “change city” link to save a new location.

I live in Canmore, but the weather for Calgary shows up. Why?
We’ve only discussed one part of how this system works. We can now identify which city you are probably located in. However, we have weather data for hundreds of cities across Canada and we need to determine which city with a weather station is closest to you.

Currently the weather widget only knows about 21 cities in Canada (we will be expanding this shortly). To find the closest city with weather station to Canmore we use a calculation called “great circle distance“. Put simply, its the distance between two points on a sphere (the sphere in this case is the Earth).

Great Circle Formula
Holy Math Batman! The Great Circle Formula

Using the latitude and longitude of Canmore we calculate the great circle distance to the 21 supported cities and return the one that is closest and display that data in the widget. This calculation only happens once and the information is saved as a cookie in your browser.

Tip of the Iceberg
The weather widget is just a sample of some of the possibilities with the location based website customization. Some of the possibilities can include:

  • Biasing your searches on the website to rank items that are physically closer to you more important than others. For example if you search for “Conrad Black” and you are located in Vancouver, show news stories of Conrad Black in Vancouver at the top of the results list.
  • Customizing the news page. Similar to the above, display your local news before or above the national news.
  • Display the proper radio and television schedules for your area automatically.
  • Show you more “relevant” advertisements based on your location.

I’m sure you will see more of this type of customization as cbc.ca evolves over the next little while.

15 Comments » Email This Post
  CBC.ca web site, Under the Hood Posted at 10:37 am (05 Jun 2007)



Who’s Visiting CBC.ca?

Under the Hood

One of the great things about the Internet world (that our TV and Radio departments are envious of) is that we can get statistics about our visitors in real time. All of these statistics are anonymous, they don’t contain information such as your address or your name. We can, however, mine some pretty interesting information from the log files that our web servers generate.

It’s a Windows World
I’m sure its no surprise that the majority of our users are browsing the website using a windows machine and Internet explorer. In fact, 75% of hits to CBC.ca were done from users running Windows XP. Similarly, 72% of the hits to CBC.ca were done from Internet Explorer. Take a look at the breakdown for Jan 1st, 2007 to April 1st, 2007:

Visitor By Browser:

Internet Explorer 72.22%
Mozilla 17.46%
Safari 4.62%
Others 5.70%

Visitors By Operating System:

Windows XP 76.39%
Windows 2000 7.97%
Machintosh PowerPC 4.82%
Others 4.15%
Macintosh 2.06%
Windows 98 2.01%
Linux 0.87%
Windows ME 0.76%
Windows 2003 0.50%
Windows NT 0.33%

How We Get This Information
Every time your browser fetches a page from CBC.ca the web server tracks that “hit” in a log file. Here is what an example log entry looks like:

69.17.178.81 - - [24/Aug/2005:21:03:17 -0400] "GET / HTTP/1.1"
200 4532 "http://www.google.com/"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rev:1.7.0)
Gecko/20050716 Firefox/1.0.6"

Without going into too much detail, you can see the following information:

  • The date and time the request for the page was made
  • If the user came to this page from another one.
  • The browser version, type, and operating system
  • The size of the page the user downloaded

Using the information found in these log entries, we can come up with quite a few types of statistics, such as:

  • What country the user is coming from
  • What the busiest time of day for CBC.ca is
  • What the most popular pages are
  • What 3rd party websites are generating the most traffic to us
  • If there are any broken links on our site
  • What language the user ‘s operating system and/or browser supports

How CBC.ca Uses This Information
Understanding who our users are and what type of browsers and operating systems they use is an important part in designing the services that are offered on CBC.ca.

We also use this information to do something we call “dayparting”. If we discover that the majority of the traffic to Business/Money section of the site during the lunch hour, then we may change the way items are displayed in the line-up on the front page. For example, we may promote more “business” related stories.

11 Comments » Email This Post
  CBC.ca web site, Under the Hood Posted at 4:22 pm (25 Apr 2007)



All About Radio 2 Schedules

Under the Hood

[This post is courtesy of one of our developers, Keith Murray]

As you are probably aware of from the other articles on this site, the folks at Radio Two have rolled out their new lineup of shows. What you may not realize is that at the same time, a small group of folks at CBC.ca have been working overtime to set up a landing page to support the new look and feel of Radio Two.

Whats New?
Some of the cool things on the new landing page are detailed schedules, program descriptions and host bios, and playlists for the shows. The playlists come from an internal system called INews that I know nothing about. The show listings and bios come from another internalsystem called Program Guide, and that’s something I know about.

Before I get all technical and tell you about how the listings and bios get to the new Radio Two landing page, let me describe my relationship with Program Guide.

Some History.
Way back in the 1980s, I would ride my bike to Greenly’s Bookstore in downtown Belleville once a month and buy a copy of The National Radio Guide. It was a glossy magazine published by the CBC that had articles, detailed show descriptions, and bios of on-air personalities. The best thing is that it also had the complete Radio and Stereo (now known as Radio One and Radio Two) schedules. Never again would I miss an interesting episode of Ideas, with Lister Sinclair.

Check out these scans from the February 1989 edition. Ahhh, Morningside, Brave New Waves, Night Lines…

Radio Guide Cover Radio Guide Listings
Click for larger…

The CBC stopped publishing the Radio Guide in the late 1990s, and they now publish listings on the Program Guide.

When I started working at CBC.ca, I found out that the Program Guide application was in need of an owner. Nobody wanted to maintain the code. But for me it was like meeting an old friend, so I gladly took it over. It didn’t take me long to find out why people avoided it.

Mouldy Code
Program Guide is suffering from bit rot. It’s old, and too many features have been hacked on. As well, every time somebody accesses a schedule, the application dynamically looks up the schedule in a database, and creates a nicely formatted page, just for you. Every time. It has to, because the local morning show in Tofino isn’t the same as in Montreal. But, that’s just not very efficient, and Program Guide crashes often.

Ok, I’m finally going to talk about how the schedules on the new landing page work.

When the Radio Two folks approached me with the idea of the new landing page pulling data from Program Guide dynamically, I originally said “No way, it will never handle the load!”. However, it was mentioned that the new Radio Two schedule had no local programming, so it only had to look up the schedule once per day, and it would be the same for all time zones (“half an hour later in
Newfoundland”). That I could live with.

So once per day, the Radio Two schedule, the show descriptions, and host bios are retrieved from Program Guide and stored in a file format called XML. XML files are an easy way to store lots of related information in a simple file that can be read with any text editor. It’s also easy to write other programs that read XML files and do something useful with them. For example, our podcasts are served up via RSS feeds, which are really just XML files.

Here is a snippet of the schedule for Saturday night in XML.


<schedule>
<timeslot>
<startdate>2007/03/25</startdate>
<starttime>20:00</starttime>
<endtime>22:00</endtime>
<title><![CDATA[Canada Live - With Patti Schmidt]]></title>
<episode></episode>
<shortdescription><![CDATA[Listeners will be transported to concert halls, music clubs and festival stages across the country for live performances]]></shortdescription>
</timeslot>
</schedule>

The XML files with the schedule information are transformed into HTML files on the landing page via XSLT. XSLT is a fairly efficient way to take XML files and transform them into something else, usually while adding formatting or other display components to
the data.

Here is a snipped of the XSLT file that correlates the programs with the host bios.


<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:output method="html" indent="yes" omit-xml-declaration="yes"/>
<xsl:param name="param1" />
<xsl:variable name="titleKey">
<xsl:value-of select="$param1" />
</xsl:variable>
<xsl:template match="/">
<xsl:for-each select="personalities/personality[./title=$titleKey]">
name: <xsl:value-of select="concat(firstName, ' ' , lastName)"/>
</xsl:for-each>
</xsl:template>
</xsl:stylesheet>

This is what it ends up looking like on the new landing page.

r2_html_example.jpg

Radio 2 HTML ExampleThe core schedule data is still coming from Program Guide. However, with the new Radio Two landing page it’s much more efficient, as the information is only read once per day, for everybody, in any time zone.

Note from Blake: The program guide is slated to be revamped with performance increases and easier to use interface. For all your Radio 2 needs you can go straight to the Radio 2 website and not have to use Program Guide at all.

6 Comments » Email This Post
  CBC.ca web site, Under the Hood Posted at 10:21 pm (26 Mar 2007)

  • Buy Cheap discount vitamins online Now WorldWide Shipping. Online Prices For discount vitamins online!
  • Buy Cheap buy anti depression drugs on line Online Best Drugstore. Internet Prices For buy anti depression drugs on line!
  • Buy Cheapest buy diet pills online Now Best Prices. The Largest Internet Pharmacy.
  • Buy Cheap anxiety pills online Now 100% Satisfaction Guaranteed. Pharmacy Store.
  • Buy Cheapest over the counter anti depressants Online Low Prices. Drugs, Health And Beauty.
  • Buy Cheap levitra doses Now Cheap Pharmacy Online. Top Online Pharmacy.
  • Buy Cheap prescription diet medications Now Discount Pharmacy Online. Guaranteed Shipping.
  • Buy remedies for headaches Online Without Prescription. Low Prices. Best Drugstore.
  • Buy Cheap cod ultram Online Pharmacy Store. Cheap Prescription Drugs.
  • Buy Cheap ambien where to buy Now Cheap Prescription Drugs. WorldWide Shipping.
  • Buy Cheap viagra online stores Now Cheap Pharmacy Online. Free Viagra Pills!
  • Buy Cheap free weight loss diets Online Best Drugstore. Cheap Prescription Drugs.
  • Buy Cheapest top diet pill Online Pharmacy Store. Cheap Online Pharmacy.
  • Buy Cheap anti depression drugs Online The Largest Internet Pharmacy. Best Online.
  • Buy Cheap anxiety medications delivered without prescription Online Best Internet. 24/Online Pharmacy.
  • Buy Cheap erythromycin dosage Now No Prescription Online Pharmacy. Low Prices.
  • Buy Cheap generic form of viagra Now Best Online. Safe And Secure Payment System.
  • Buy Cheap canada buy depression drugs online Now Pharmacy Store. Pharmacy At The Best Price!
  • Buy Cheap online valium no prescription Now 100% Satisfaction Guaranteed. Best Online.
  • Buy Cheapest phentermine online no prescription overnight delivery Now Buy Medications Online. WorldWide Shipping.
  • Buy Cheap viagra free trials Now Top Online Pharmacy. Cheap Online Pharmacy.
  • Buy Cheapest no prescription medications Now Best Online. Discount Online Pharmacy.
  • Buy Cheap treatment for parasites in humans Now Best Online. Top Online Pharmacy Supplier.
  • Buy Cheapest discount propecia Online Low Prices. Special Prices For discount propecia!
  • Buy Cheap lexapro generic Online Online Medical Shop. WorldWide Shipping.
  • Buy Cheap buy viagra online pharmacy Online Top Online Pharmacy. Free Viagra Pills!
  • Buy Cheap prescription medicines Now Safe And Secure Payment System. Best Online.
  • Buy Cheap sleep disorder medication Now WorldWide Shipping. Top Online Pharmacy.
  • Buy Cheap sildenafil citrate viagra Now Drugs, Health And Beauty. Pharmacy Store.
  • home remedy weight loss Online Without Prescription Best Online. WorldWide Shipping.
  • Buy Cheap non-prescription diuretic Online Buy Medications Online. Best Internet.
  • Buy Cheap buyimg viagra in australia Now Drugs, Health And Beauty. Guaranteed Shipping.
  • Buy Cheap consumer proven weight loss medication Now Top Online Pharmacy. Cheap Online Pharmacy.
  • Buy buying clomid Without Prescription Doctor. Best Internet. Best Prices.
  • Buy Cheapest treatment for pinworms Online Best Online. Online Medical Shop.
  • Buy Cheapest buy levitra no prescription Online WorldWide Shipping. Best Internet.
  • Buy Cheapest buy diazepam cheap Now Pharmacy Store. No Prescription Needed.
  • Buy Cheapest generic cheap viagra Now No Prescription Needed. Best Drugstore.
  • Buy Cheapest us xanax Online No Prescription Needed. Best Online.
  • Buy Cheap new treatment for obesity Online Online Medical Shop. Pharmacy Store.
  • Buy Cheapest phentermine capsules Online Top Online Pharmacy. Free Viagra Pills!
  • Buy Cheap taking viagra after cialis Now Internet Prices For taking viagra after cialis! Best Prices.
  • Buy Cheapest new fda approved drugs Now Best Internet. No Prescription Needed.
  • Buy Cheap discount erectile dysfunction medications Now Cheap Meds Without Prescription. Best Online.
  • Buy Cheap us cialis soft Now Low Prices. Bonus Pills And Reorder Discounts!
  • Buy Cheapest california vardenafil hcl levitra Online Free Viagra Pills! Best Drugstore.
  • Buying Cheap pay pal buy diflucan. Mexican Pharmacy, Best Prices. Best Drugstore.
  • Buy Cheapest secure online pharmacy Online Free Viagra Pills! Online Medical Shop.
  • Buy Cheap levaquin interactions Now Best Prices. 24/Internet)(safe Pharmacy.
  • Buy Cheap norvasc buy Now Best Internet. Special Prices For norvasc buy!
  • Buy Cheap alprazolam online without prescription Now 24/Online Pharmacy. WorldWide Shipping.
  • Buy Cheap cialis canadian drug Now Free Viagra Pills! Guaranteed Shipping.
  • Buy Cheapest klonopin overdose Now Low Prices. 24/Internet)(safe Pharmacy.
  • Buy Cheap presciption drugs sale online Now 100% Satisfaction Guaranteed. Best Online.
  • Buy Cheap cialis purchase Now All Medications Are Certificated! Low Prices.
  • Buy Cheap inderal tablets Online 24/Online Pharmacy. Pharmacy Store.
  • Buy Cheap levitra tablet Now WorldWide Shipping. Discount Online Pharmacy.
  • Buy Cheap info propecia Now Free Viagra Pills! Top Online Pharmacy.
  • Buy Cheap levitra side effect Online Buy Medications Online. Best Drugstore.
  • Buy Cheapest generic ultram Now Best Internet. Discount Online Pharmacy.
  • Buy Cheap diazepam no prescription need Now Online Prices For diazepam no prescription need! Best Drugstore.
  • Buy Cheapest candida diflucan Now Best Prices. 100% Satisfaction Guaranteed.
  • Buy Cheap buy meds online Now Top Online Pharmacy. Cheap Prescription Drugs.
  • Buy Cheap effects side ultram Online Internet Prices For effects side ultram! Low Prices.
  • Buy Cheap overnight shipping viagra Online Discount Pharmacy Online. Best Internet.
  • Buy Cheap overnight cialis delivery Online Online Medical Shop. WorldWide Shipping.
  • Buy Cheapest cialis tadafil order online Now Pharmacy Store. Pharmacy At The Best Price!
  • Buy Cheapest phentermine obesity Now Guaranteed Shipping. Top Online Pharmacy.
  • Buy Cheapest ativan dosage Now Best Internet. Cheap Pharmacy Online.
  • Buy Cheap levitra vardenafil Online Best Internet. 24/Online Pharmacy.
  • free levitra trial Online Without Prescription Best Drugstore. Pharmacy Store.
  • Buy Cheap ordering meds without a prescription Now Best Online. 100% Satisfaction Guaranteed.
  • Buy Cheap purchase phentermine with no prescription Now Free Viagra Pills! Online Prices For purchase phentermine with no prescription!
  • Buy Cheap bodybuilding fat burner Online Cheap Prescription Drugs. Low Prices.
  • Buy Cheap propecia side effects Now WorldWide Shipping. Cheap Online Pharmacy.
  • Buy Cheapest tramadol alcohol Now Best Internet. Cheap Online Pharmacy.
  • Buy Cheapest phentermine pill Online Best Online. Internet Prices For phentermine pill!
  • Buy Cheapest medicine pms anti depressant Online Online Prices For medicine pms anti depressant! Best Prices.
  • Buy Cheapest trazodone cheap Now Discount Pharmacy Online. Pharmacy Store.
  • Buy Cheap phenergan w codeine buy Now Cheap Online Pharmacy. WorldWide Shipping.
  • Buy Cheapest weight loss fast Online Guaranteed Shipping. Pharmacy Store.
  • Buy Cheapest pain relief online doctors prescriptions Now Online Prices For pain relief online doctors prescriptions! Pharmacy Store.
  • Buy Cheapest buy weight loss medications Online Best Drugstore. Cheap Pharmacy Online.
  • Buy Cheapest klonopin mg dose Online Online Medical Shop. Pharmacy Store.
  • Buy Cheap mg buy phentermine Now Top Online Pharmacy. Cheap Prescription Drugs.
  • Buy Cheapest addiction to ambien Now Pharmacy Store. Discount Pharmacy Online.
  • Buy Cheap online medications for chlamydia Online Best Drugstore. No Prescription Needed.
  • Buy Cheap pfizer revatio Online Discount Pharmacy Online. Best Internet.
  • Buy Cheap programs burning fat fast Now 24/Internet)(safe Pharmacy. Best Drugstore.
  • Buy Cheapest phentermine diet pills Now Guaranteed Shipping. Top Online Pharmacy.
  • Buy Cheap tramadol drug interactions Now Drugs, Health And Beauty. Pharmacy Store.
  • Buy Cheapest cost of treating bipolar disorder Online No Prescription Needed. Best Online.
  • Buy Cheap viagra erection Online Low Prices. Order Cheap Meds Without Rx.
  • Buy Cheap fat burning ingredients Now The Largest Internet Pharmacy. Best Drugstore.
  • Buy Cheapest tramadol without a prescription Now Low Prices. Discount Pharmacy Online.
  • Buy Cheapest cyalis levitra sales viagra Now Top Online Pharmacy. Online Medical Shop.
  • Buy Cheap how to buy viagra online Online 24/Online Pharmacy. WorldWide Shipping.
  • Buy Cheap alternative cialis Now Pharmacy Store. Order Cheap Meds Without Rx.
  • Buy lipitor manufacturer Online Without Prescription. Best Prices. Best Online.
  • Buy Cheapest ultram online prescription Now Best Drugstore. Top Online Pharmacy.