Follow up: so I connected the Ethernet cable to a USB 3.0 to Ethernet adapter and plugged it in to the TV's USB port.
My download speed went from 95 Mbps to 346 Mbps. The reason it didn't go higher (it's a 1 Gbps fiber connection) is, I think, because the TV uses USB 2.0 version in its ports, which aren't capable of 1 Gbps speeds.
But for $14 I essentially more than tripled my download speed.