16:16 <imaximets> Hey! It's time for the weekly OVN meeting.

16:16 <fnordahl> o/

16:16 <mkalcok> o/

16:17 <imaximets> Mark is not here today, so I guess I'll host.

16:17 <haleyb> o/

16:17 <imaximets> I'll start with a short update from my side.

16:17 <imaximets> Didn't really do much OVN-related, been focusing on reviews for OVS branching.

16:18 <imaximets> Created branch-3.6 yesterday.

16:18 <imaximets> So, please, test and report issues!

16:18 <imaximets> That's it from me.

16:18 <imaximets> Who wants to share next?

16:19 <mkalcok> For me it was mostly downstream stuff, so I'm in ro mode today :)

16:19 <imaximets> Ack. Thanks, mkalcok.

16:19 <imaximets> Anyone else?

16:19 <zhouhan> I can go next

16:20 <imaximets> zhouhan, sure.

16:20 <zhouhan> I sent patches for the HW offload fix for review: https://patchwork.ozlabs.org/project/ovn/list/?series=465368

16:20 <zhouhan> I also merged the long-hanging patch for lowering the priority for src-routes

16:20 <zhouhan> That's it from me

16:20 <imaximets> Thanks, zhouhan !

16:21 <haleyb> i have one item, more a question

16:21 <imaximets> haleyb, go ahead.

16:22 <haleyb> not sure if it was raised previously, but we are seeing an issue with OVN/OVS when using DPDK

16:22 <haleyb> UDP fragments are not making it through

16:22 <haleyb> https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2115795

16:24 <imaximets> Have you tried newer OVS/OVN?

16:24 <haleyb> imaximets: yes, we have tried a later version, ovs 3.5 i think

16:24 <zhouhan> imaximets: does userspace datapath conntrack support IP fragments?

16:25 <imaximets> One thing to point out is that OVN depended on kernel reassembling fragments for a while. not sure if that was fixed in latest.

16:25 <imaximets> And this is not how userspace conntrack works with DPDK.

16:25 <fnordahl> There was this fix https://patchwork.ozlabs.org/project/ovn/list/?series=457629&state=* but it appears to only address the LB use case

16:26 <zhouhan> fnordahl: yes, and that one broke HW offload. My fix was for that :)

16:26 <fnordahl> It's strange how its all connected somehow!

16:27 <haleyb> zhouhan: your question about 'is it supported' is my high-level question as well, i don't know

16:28 <imaximets> Either way, I'd suggest to look at places where connection tracking is happening and there is potential for matching on L4 fields, as there might be some other places that are similarly broken.

16:28 <imaximets> fragments are supported, but there are differences in how conntrack treats them.

16:29 <imaximets> After ct() action in the kernel, the packets are re-assembeled, so you can match on L4.

16:29 <haleyb> and we think we were able to workaround it manually, i just added a comment to that bug, but i'm not knowledgable enough regarding ovn-northd on where to look

16:29 <imaximets> But after ct() action in userspace, the packets are still fragmented, and you must match on ct metadata instead of L4 fields.

16:30 <imaximets> Technically, userspace implementation is true to OpenFlow, but we can't make kernel conntrack to not reassemble...

16:30 <imaximets> OK.

16:31 <imaximets> Thanks, haleyb for bringing this up. Maybe someone can take a closer look later.

16:31 <imaximets> Do we have anyone else who wants to share today?

16:32 <haleyb> imaximets: ack, thanks, even if just a pointer on where to look, i'm more the openstack expert :)

16:32 <fnordahl> Oneliner for me, participated in the upstream Neutron native OVN BGP spec meeting, otherwise mostly downstream stuff and I still owe a few reviews which I have not forgotten.

16:32 <elinux> can someone look into  https://bugs.launchpad.net/networking-ovn/+bug/2116942 please if possible ?

16:32 <zhouhan> haleyb: is the first UDP packet fragmented already?

16:33 <haleyb> zhouhan: yes, but i believe it has the entire UDP header, just not the second fragment

16:34 <zhouhan> For tcp, usually the first packets for hand-shaking are small (not fragmented), and once connection is established the L4 header is not checked. But for UDP, if the first packet is large and fragmented, probably the second fragment is dropped because the connection state never gets to est ...

16:34 <haleyb> elinux: the networking-ovn project has been retired, we need to add neutron there and add the ovn tag

16:35 <haleyb> zhouhan: yes i believe that is the issue

16:35 <elinux> posted this https://bugs.launchpad.net/neutron/+bug/2117078

16:36 <imaximets> zhouhan, haleyb, the second fragment will be +new, and the ct metadata should have the L4 info.

16:36 <haleyb> alright, should have just added neutron but someone will triage

16:36 <haleyb> sorry, that was for elinux ^^

16:36 <zhouhan> imaximets: understand, so the ACL pipeline needs similar changes

16:37 <imaximets> Probably.

16:37 <elinux> the logical flows are not getting created in the SB db as soon as the sriov based external ports are created

16:38 <imaximets> OK. Ack, elinux, I guess, someone from neutron should take a look at your issue.

16:39 <haleyb> elinux: someone will triage that neutron bug, maybe me later, or our weekly deputy

16:39 <haleyb> we have meetings every tuesday on OFTC

16:39 <elinux> thanks folks ...

16:39 <imaximets> zhouhan, haleyb, there are might be other cases indeed where OVN mistakenly assumes that packets are not fragmented after ct().

16:40 <imaximets> I suppose, one way to confirm is to look for datapath flows that match on all-zero L4 fields and have a drop action.

16:40 <imaximets> Hopefully, that's enough of a hint. And I'm not sure we can guess-debug here further.

16:40 <haleyb> imaximets: understood. if there is any debugging you think is useful let me know

16:40 <zhouhan> imaximets: I took a note for this issue :)

16:41 <imaximets> zhouhan, thanks!

16:41 <haleyb> and not sure if there is a better bug tracker than launchpad

16:41 <imaximets> Anyone else wants to share?

16:42 <imaximets> I guess, that's it for today.

16:42 <imaximets> That was a productive meeting. :)

16:42 <imaximets> See you next week.

16:42 <fnordahl> \o thanks all cheers!

16:42 <mkalcok> \o see you :)

16:42 <imaximets> Buy!

16:43 <imaximets> *Bue. :)

16:43 <imaximets> *Bye.... :/

16:44 <haleyb> o/