Had a strange issue with Verizon FiOS; connected my switch (ports in VLAN 3) to the 100MB Ethernet drop they hand-off. Everything worked fine for the first device I connected, but when I tried to add additional devices, no-go, not even a response to an ARP from Verizon's gateway. Finally, after having numerous tech support calls, a replacement of the ONU/ONT, I got their tier2-3 support on the phone and they managed to reset the ONU/ONT after a couple of hours. A power cycle didn't do it, they had to do something special. Apparently the issue was the MAC table on the ONU/ONT, although they swear there is no limit (*cough*). The issue with their equipment was actually exacerbated by the excessive number of MAC addresses I advertised, primarily due to the SMC switch issue (below), apparently their box stops accepting MAC addresses after about 25-30. Since I had purchased service for 5 IP addresses that should have been fine.
To fix on SMC switch: Open the web GUI and navigate to 'VLAN/802.1Q VLAN/static members'. This is where I figured out that the switch was actually putting frames from vlan 1 on the other VLANs. However, since the new native untagged vlan on these ports (vlan 3) was not tagged, the VLAN 1 traffic was tagged and went nowhere. This just wasted bandwidth, and more importantly put about 30 extra MAC addresses in the table of the Verizon equipment (ONU/ONT). This was apparently enough to make their box stop accepting traffic from new MAC addresses. meaning that I couldn't even ARP to their router. Also important to check the SMC boxes on this section: 'VLAN/802.1Q VLAN/static members' and put 'forbidden' to prevent this from happening again. Basically I put forbidden for each vlan on the ports that it shouldn't be on. Every port is either set to 'untagged' (in the vlan as normal) or as 'forbidden', and do this for each vlan. As far as I could find there is nowhere to forbid 802.1Q on ports, you can only set it to 'ON' (tag everything), or to 'hybrid' (tag all but native). It's nice that it has these fancy options, but how about an off switch?