KVM Forum 2011

This year’s KVM Forum, like last year’s, was co-located with LinuxCon NA.  Vancouver city played the host this year.

The interest in KVM has been rising over the years; from the first Forum in 2008, when we were just about 30 developers in a single room presenting work done and chatting about directions to take (the virtio design was hashed out during this conference), this year there were about 150 attendees, discussing optimising KVM instances and tracing the guests.  That’s a really big leap in three years.

Due to lots and lots of good talk submissions, not all of which could be rejected, the talk slots were reduced to 30 minutes per talk and there were parallel tracks in the afternoon sessions.  This allowed for more talks, but small Q&A sessions, and obviously, having to miss out on some talks due to another talk happening at the same time. All the talks have been video-recorded, though, and they should appear soon on the Forum page.

Some brief notes on the talks I attended:

Avi’s keynote started off the Forum. One of the main points was the lack of marketing in KVM, and how the formation of the Open Virtualization Alliance to bridge the gap.  He also talked about an ARM port finally getting some development (on his 2008 prediction of an ARM port coming soon not materialising, he said in his 2010 keynote: “this is a case of reality not catching up with predictions”).

Paul Mackerras then talked about KVM on the POWER7 processor, another processor to get virt extensions. He talked of replacing pHype with KVM and the challenges to run custom firmware and Linux directly on the machines, as opposed to the default firmware which can only run pHype.

Alex Williamson talked about VFIO-based PCI device assignment.  The current device assignment code makes the kvm.ko module a device driver for the device to be assigned to the guest (I’m to blame for that, I wrote that code). The idea with VFIO is to move the complexity of device assignment into userspace.  VFIO is a device driver which exposes devices in /dev/vfio*, via which the device can be configured and controlled.  This is a much more cleaner and secure way of doing device assignment.

Kevin Wolf talked on the current state of block file formats, and the next-gen block file format.  He criticised the NIH syndrome of people developing new formats in isolation instead of enhancing the current ones.  He’s working on collecting the best ideas from QED and FVD, the newest formats, and putting them into QCOW3 while retaining the features from QCOW2, which the other formats dropped.

Stefan Hajnoczi and Paolo Bonzini then talked of a new virtio-scsi transport, Stefan is working on the new in-kernel SCSI target and using vhost to accelerate communication.  Paolo has already written the virtio-scsi spec.

Asias He presented the Native Linux KVM tool. My reaction on the presentation was that they started out as a toy project to run Linux guests, but their planned feature set sounds like they are going to replicate qemu.  That’s not a bad thing, though, KVM (the kernel module) was designed to be able to drive multiple userspace hypervisors, and this is the first one that’s making some news.

That ended the first day’s morning session.  The afternoon session had parallel tracks, I attended the following:

Andrea Arcangeli’s talk on the future of Memory Management in KVM had quite a lot of TODO items.  He particularly talked on NUMA management and his ongoing work on it. Current NUMA policies are static; he wants to make them dynamic, with the guest moving to the node where RAM is allocated, and vice-versa.

Rik van Riel then talked about some more MM work: free page hinting, which can improve the memory utilisation both in the host and guest, and automatic memory resizing.  There might be some drawbacks to this as free page hinting may not consider THP and end up breaking huge pages.

Next was “experiences porting KVM to SmartOS“, a lively and animated talk by Bryan Cantrill. This talked about porting the kvm module and qemu-kvm to Illumos, a Solaris clone.  They primarily want the benefits of ZFS, DTrace, Zones and KVM. No matter how much interesting it sounds, the question on licensing was addressed vaguely (if at all) during the talk.  In a private chat later, Bryan mentioned there’s no violation at all.  There’s some talk at lwn.net on licensing as well.

Michael Tsirkin talked on new virtio networking features. Main was the event indexing feature which reduces the exit interrupts to host if there are pending exits and a new buffer is queued in the vring.  Sort of like NAPI for virtio. He also talked about zero-copy TX and filtering, and the security pitfalls of doing so.

Ryan Harper then talked of IO throttling in QEMU, a feature that uses cgroups to ensure guests don’t go over their allocated quota of IO activity.

A couple of lightning talks were held, where Dan Magenheimer talked of Transcendent memory, and how that can help with the work that Rik is doing.

A few BOF sessions were lined up, people gathered in groups to discuss.  I caught hold of Hans de Goede, Alon Levy, Anthony Liguori and Gerd Hoffmann to discuss the state of chardevs in QEMU.  Hans had initiated a discussion just prior to the Forum on the non-upstream RHEL and Fedora patches that we carry for chardev flow control.  Anthony mentioned some races in the existing implementation and came up with his own.  He promised to merge the cleanup patchset soon and float the flow control patches to the mailing list.

My other topic, on guest – host communication, got fizzled out, partly due to my jet lag not allowing me to concentrate much, and the other interesting topic, moving qemu away from C.  I used that time to talk with other people.

That ended day 1 of talks. All the attendees then headed out to a pub nearby to exchange stories over beer.

The second day started with Anthony Liguori presenting the keynote on QEMU development. He mentioned how the project has been doing very well with sub-maintainers doing pull requests. A lot of patches have been committed since the last year. Things indeed have improved since the last year, when many people were complaining of patches bit-rotting on the mailing list for ages.

Avi then took stage again to talk of performance monitoring in KVM guests. He talked of providing a Performance Monitoring Unit to the guest via several ways: pass-through, emulating a virtual PMU and emulating a real-life PMU. He also talked of some new PMU features which are not model-specific which can be safely exposed to all guests.

Alex Graf then presented on AHCI.  This was a very cool presentation with nice animation effects (too sad it used non-free software to do that — I don’t know if free software can match those effects, though). He showed how AHCI performed much better than the default IDE storage type.  Performance is half-way between virtio-blk and IDE, but since most OSes support AHCI out of the box (notable exception being Windows XP), he made a case for making AHCI the default.  There is some work to be done before we can do that, though.

Anthony Liguori next talked about QAPI and QOM, the QEMU Object Model.  These refactorings will make QEMU machines much easier to generate, and present a much saner interface to higher-level management tools like libvirt.  The plan is to get as much work done for the impending 1.0 release.  It was refreshing to hear Anthony not talk of replacing code in one big patch (or one big series), and rather work in incremental steps in-tree.  His last year’s main point of developing code in separate trees and doing merges had not gone down well with many developers.

Markus Armbruster then talked on qdev, on where we are, what’s left, and what are the major pain points.  qdev conversion still remains one of the TODO items from last year, and the more it gets delayed, the more everything else gets delayed in QEMU (including QOM conversion, which could be an incremental step from qdev).

Alon Levy then presented on SPICE, the current status as well as the future. The SPICE protocol is an alternative to VNC with a much better focus on high-latency links and more than just video over network.

Gerd Hoffmann described his work on the USB subsystem. QEMU could go from the last project to support USB 2 to the first one to support USB 3. He also highlighted the work done on bringing down the CPU usage with USB tablet devices to minimal, a common complaint that was heard from users.

That ended the first session; the talks I attended in the parallel tracks were:

KVM Graphics Device Assignment by Allen Kay. We had worked together on PCI device assignment a few years back, and now Allen Kay talked of some roadblocks and ideas in implementing graphics device assignment and experiences from doing so in Xen.

Live block copy in QEMU is being worked on by a few people, Marcelo Tosatti presented the work done so far and the direction in the future. He talked of how the two seemingly independent features of live block copy and snapshot merges can share code.

Joerg Roedel then talked of AMD IOMMU v2 support in KVM: the new feature set makes it possible to not pin all the guest memory pages on the host.  This alone is a very important feature for the future of device assignment.

Next up was Juan Quintela’s session on Live Migration.  It was an entertaining ride on the challenges faced and the new directions to take.  One of them was post-copy migration, where the guest memory is faulted over the network after the guest is running on the destination host, since the amount of guest RAM has been increasing over time.

I missed out on the next two sesions, talking to people.

I rejoined for postcopy live migration by Takahiro Hirofuchi. As promising as it may sound, Anthony wanted to ensure we have eked out maximum performance from the current pre-copy implementation and then look at post-copy.  He also asked for benchmarking results for post-copy migration.  An interesting case here may be to guess the working set of a guest, perform a pre-copy using this set of pages, and then switch to post-copy.  The guessing of working set could be done via a guest agent or using MMU notifiers in the host.

That ended a very very long two days of the KVM Forum. We Red Hat folks had a dinner hosted by CTO Brian Stevens, so we headed out to the nearby brewery and enjoyed the fresh lager there.

FUDCon India F2F Planning Meeting Minutes (Jul 26 2011)

In this edition of the planning meeting, Suchakra joined us at the Red Hat Pune office.  The others present were Rahul, PJP, Sathya, Shreyank, Saleem, Shakthi and me.

Having Suchakra attend the meeting was great, we made some good progress on the venue-related items that we needed some greater clarity on.

He clarified the bandwidth would be sufficient for our usage, identified a few students as volunteers (we will meet them soon), mentioned setting up a public Fedora mirror would not be difficult at all (bandwidth and storage are amply available).  All this is great news, and he’s also going to be around for the next 2 months to co-ordinate with any other venue-related matters.


  • FUDCon website
    • Owner: Saleem
    • Most work has been done
    • Needs help from Rahul, Suchakra on content
    • Needs help from Tatica on design
    • Final website will be at: http://fudcon.fedoraproject.org
  • Catering
    • Owner: PJP
    • Got couple of more contacts from Murty
    • Will follow up with them this week
  • Hotel
    • Finalising between two options: Cocoon (within Magarpatta City, very close to Red Hat office) and Royal Orchid in Kalyani Nagar
    • Costs $40 (Cocoon) vs $90 (Royal Orchid)
    • We have a better deal with Cocoon: in addition to a special rate, we are also getting
      • laundry
      • breakfast
      • wi-fi
      • mineral water
      • better rate for FUDPub venue
  • Bus
    • More than one may be required depending on people
  • FUDPub
    • Don’t know rates yet
    • Will be decided this week
    • Min. legal drinking age in Maharashtra is 25.  How to filter people?
    • Microbrewery can supply beer at our chosen venue.
      • Deal has to be worked out such that we pay only for what we consume
  • Information on website (plus booklet)
    • Photos
      • Cocoon (accomodation)
      • Magarpatta city
      • FUDPub venue
      • COEP (event venue)
  • Videos
    • Shreyank didn’t receive input from Shravan at COEP
    • Rahul to follow up with PyCon people
  • Artwork
    • 3rd week of August deadline for 1st draft of all designs
      • t-shirts, posters, banners, website, etc.
  • Bandwidth
    • PJP hasn’t spoken with BSNL people yet
    • Suchakra mentions bandwidth at venue is sufficient (with failover)
      • Recent MITConf served 100 people without glitches
  • Volunteer team
    • Shravan will be heading team
    • Shravan handles routers and infrastructure at COEP
    • Suchakra and Shravan to perform a mock install
    • Saleem and Shravan to run a test install 1 month prior to event
      • Checking router capacity, bandwidth, plug points, etc.
  • Fedora Mirror
    • Owner: Suchakra, Rahul, Saleem
  • Canteen
    • Already open on Fri/Sat. Suchakra will confirm if they can stay open on Sunday
    • Can we host outside caterers at the venue (canteen or outside)? – Suchakra to confirm.
  • Scheduling FADs before FUDCon
    • Owners: Rahul, Shakthi
    • Oct 1st week and Oct end won’t clash with exam schedule
    • Suchakra says we can expect about 60 people
    •  Talk to Prof. Abhijit about this — Suchakra + Rahul
    • These FADs will be prep sessions for the FUDCon
      • Introduce Fedora terminology
      • Introductory / basic material as prelude to talks at FUDCon for the benefit of students
  • Sponsorship requests
    • Rahul officially opened sponsorship window.  Requests pouring in.
    • Deadline of Aug 5, can be extended depending on budget.
    • Review of requests will be done after the deadline
  • Content
    • This now takes 1st priority
    • Rahul to update wiki with hotel info
    • COEP pictures to be uploaded by Suchakra
    • Fill in all details; booklet data can be scraped from website.
  • COEP
    • Organisers to visit venue and meet student volunteers, prof Abhijit sometime in the next 1-2 weeks
    • Figure out which rooms are being allocated to us, check distance between them
    • Power point availability in the auditorium, labs and talk rooms
    • Backup power status
      • Projectors, wifi continue to work in case of power cut?

FUDCon India 2011 F2F Meeting 2

The second face-to-face meeting for the FUDCon India planning happened on Tuesday, July 19 at the Red Hat India office.

Shreyank, Saleem, Rahul, Satya, PJP, Kushal, Shakthi and me were present.

Discussion took off from where we were last time and what still needs to be done.  In addition, we now have tentative dates to get feedback on all outstanding items:

  • Confirmation on speakers flying in from outside India (who we are sponsoring)
    • Owner: Rahul
    • Few people have applied for sponsorships, including Kital, Eugene Teo, Buddhika Kurera.
    • We have to get word out to these and others who are considering flying in to book flights ASAP to get better deals.


  • Not everyone is blogging about the work they’re doing.
    • In addition to keeping everyone apprised of the situation, also helps future FUDCon planners to see what’s involved.

  • 1st priority: book hotels.
    • Murty is talking to vendors for deals.  He will have concrete answers by Thursday Jul 21st.


  • Visas
    • Items we can give guidance on / items required for obtaining a visa:
      • India gives visa on arrival for nationals of 16 countries.
      • Duration of stay?
      • Who can vouch for the individual?
      • Where will he/she stay?
      • Conference visas and tourist visas might be different for some countries
      • Ppreventive medication may be required.
    • Items we have to assist with:
      • Invitation letter
    • No progress on this till we get flights and hotel bookings sorted out.


  • Goodies
    • Owner: Kushal Das
    • Deadline: 31 Aug
    • Talk to Suchakra to co-ordinate posters and vendors.
    • Kushal will get estimates for T-shirts, posters, banners, buttons, stickers, lanyards (tags).
    • Tatica, bckurera working on design of t-shirts
    • Rahul will follow up with designers


  • Posters
    • Owner: Suchakra
    • Deadline: 31 Aug
    • Good progress; designs on India list
    • Co-ordinating with design team on logo guidelines and usage


  • Booklet
    • Owner: Ankur Sinha
    • Deadline: 31 Aug
    • Not much progress
    • Some details added, but many more needed
    • (Should mention speakers should be able to talk if projector conks off)
    • Rahul to follow up and write content


  • Caterers
    • Owner: PJP
    • Deadline: 10 Aug
    • Boxed lunches: INR 350-400 per box
      • Yellow Chillies
      • Makkhan Maarke
    • Buffet INR 75 lesser than that
    • PJP following up with other vendors


  • FUDPub
    • Owner: Kushal
    • Deadline: Kushal to have updates next week


  • Website + voting
    • Owner: Saleem
    • Deadline: next week (content, funnel, COD)
    • Saleem to set up own server / hosting
    • Kushal to look at funnel
    • Saleem looking at COD
    • Rahul doing content
    • No domain yet from Fedora Infrastructure — blocker!


  • Bandwidth
    • Owner: PJP
    • Deadline: next week
    • PJP to talk to BSNL for a wired line to the venue for about a week


  • COEP
    • Owner: Suchakra
    • Infrastructure
      • New routers added, can handle 400 users (40 users per router; 10 available)
    • Set up a mirror on campus — Rahul + Saleem + Shreyank
    • We need a student volunteer team to interact with – Shreyank to identify
    • One lab running Fedora (dual-boot should also be OK) — possible?  needed?
    • Canteen
      • Will they remain open on all days?  Serve food for people we’re expecting?  — Saleem
    • Need to identify one room as speaker’s lounge
      • Have water/tea/coffee/snacks available


  • Backup plans
    • 3G dongles for speakers (if we can’t get a dedicated ‘net connection)
    • Spare projectors if some go off


  • Misc.
    • Registration desk — identify people who will welcome people and distribute swag
      • Swag distribution can be held off till 2nd day
    • Volunteers necessary to guide people from Mumbai to Pune?

KVM Forum 2011

The KVM Forum for this year is happening in Vancouver, Canada:


The schedule’s jam-packed; this is the first time we’re going to have parallel tracks (which sucks, since one can’t attend all the talks).

There has always been interest from developers in KVM, being the nice, clean, modular solution that it is. But now that it’s been dominating the SPECVirt results, businesses are looking at it as well, so there are a few end-user talks on the schedule this time, which is nice to see.

I’m going to be running a BoF session on guest agents and guest-host communication protocols.

I’m looking forward to meet the team.  I’m not staying back for the co-located LinuxCon Vancouver (though KVM Forum participants get a free entry; so you could register for KVM Forum instead of registering for LinuxCon if you haven’t done so yet).


FUDCon India Planning: Weekly Meetings

If you want to keep track of how FUDCon India planning is shaping up, or want to lend a helping hand in deciding how it shapes up, there are two weekly meetings you should be aware of:

  1. The IRC meeting on #fudcon-planning on irc.freenode.net every Friday at 1300 UTC / 1830 IST.
  2. The weekly face-to-face meeting at the Red Hat office in Pune, India.  This happens every Tuesday at 1500 IST.  If you can’t attend these and if some other day works for you, drop us an email at the India list.

The preparations for the FUDCon are well under way, watch the India list for minute-by-minute details.

First FUDCon India meeting

A few of us at the Red Hat Pune office strolled into a conf room to discuss plans and fix responsibilities for organising the FUDCon India 2011 at Pune. It was an impromptu session; we couldn’t include non-RH organisers but we kept the details to what the people present in the room have already committed to, according to the list of organizers.

We agreed on a set of action items.  There are no dates attached yet; those should be discussed and decided in the next meeting.  Some clarity needs to come from others; the first planning meeting on IRC scheduled for the 15th of July should shed light over those matters.

In addition to all that’s listed below, we might get extra sponsorship money (in addition to the FUDCon budget) from some companies.

If there’s some budget surplus, who’d like a fully-sponsored elephant ride through the city?  (Or at least to the event venue?)  Be quick to nominate yourself!

  • All: blog about the activities you’re doing. Ensure your blog is aggregated on planet.fedoraproject.org.
    • (Rahul to contact other organisers to do this) 
    • Done: Rahul sent an email to the India list.

  • Banners – Suchakra taking care of this
  • Booklet – Ankur Sinha with others.
    • Rahul to take part in this too

  • 1st priority: get international speakers to book flights ASAP. Get them to submit ‘sponsorship needed’ tickets.
  • 2nd priority: contact HR, get invitation letters for foreign delegates
  • 3rd priority: book hotels

All these 3 depend on:

  1. who’s coming?
  2. who are we sponsoring for flights/hotels (visas are self-sponsored)
  3. other guidelines — should get clarified this friday on irc meet

  • People not listed on talks page but will come:
    • Members from the Red Hat Community Architecture team
    • They will come from their own budget. Confirm if stay is also from their budget.
    • Get them and their talks listed on Wiki page.

  • T-shirt design: Rahul to contact design team
    • Done: Rahul opened a ticket
  • Videos: Ramki to contact pycon people who have offered to videograph + host videos

  • Swag: Can come from Ambassadors budget, but we could put our money if we have enough sponsorships.
  • FUDPub: Kushal to contact pubs.

  • Lunch: Should we sponsor till a cut-off? All? Only speakers + outstation delegates?
    • Only sponsor speakers + outstation delegates, have for-pay counters for everyone else. Rahul says he hasn’t seen a conf where food is free.
    • If we get lunch out of the budget, we can do better things for fudpub (starters in addition to one round of drinks) and swag (for more people)

  • Website + online voting (for barcamp): Saleem
    • Rahul to send initial mail about website to fedora advisory board.
    • Done: Rahul sent initial mail.
  • Hotels + logistics: Satya + Murty
    • Get quotes
    • Book as soon as we know number of int’l / non-Pune delegates who we are sponsoring
    • We could book for people who are staying on their own budget, eg., self-sponsored delegates / speakers. 
  • Food/catering: PJP
    • multiple options
      • boxed set
      • stall
    • explore both for two days. 3rd day will be paid for by individuals (or adjusted if budget permits).
    • Check with COEP if they can keep canteens open for all three days
  • To check with COEP – Logistics team + Rahul
    • infrastructure – wireless
    • canteens remaining open for all 3 days
    • stalls for food allowed near conf rooms?
    • set up a fedora mirror
  • Barcamp voting
    • Need to have printers, stationery at site on first day.
    • Stick schedule per room and per day on each room.
    • Possibly display schedule on projector
    • Suggestion: Have 2-3 keynotes (talks w/o parallel tracks) on first day and ask participants to vote online or on a board somewhere before breaking for lunch. This can reduce confusion.

  • Fudcons generally have 4 parallel tracks
  • Keep a schedule ready 2 days before event; minor changes allowed after voting on first day.

FUDCon APAC 2011: Pune, Nov 4-6

Jared Smith, the Fedora Project Leader, has announced the Pune bid has won for the APAC FUDCon for 2011.


If you’re planning to attend, there’s information on travel and costs on the bid page above.  A few community volunteers who will speak at the event can be sponsored, subject to budget restrictions.

Make sure to get your proposed talks or hackfests listed on the link above.  We already have a healthy list of topics; I’m eagerly looking forward to the event.

For people local to Pune, you can help organising the event. Please contact Rahul Sundaram, the event owner, or send an email to the fedora-india mailing list for details.