d*******r
发帖数: 3299 | 1 请问板上的 Python 大牛们 (比如公孙大神), 用 Python 做cloud上的VM管理 (比如
AWS 上大量 EC2 instance 的配置和monitoring), 主要需要用熟哪些工具?
我研究了下, 关于auto deploy/cfg 的工具, 选择有 SaltStack 和 Ansible.
据说是 Ansible 更简单, 跑在 ssh connection 上, 而且Vagrant 官方支持.
SaltStack 需要安装 client 到被管理机器上, 而且 SaltStack 反馈速度更快, 跑在
ZeroMQ connection 上.
不知道 Zeromq 这种东西靠谱不, SaltStack+Zeromq 到底坑多不多?
看 SaltStack 官网, 现在也有 Salt Air 了, 也可以模拟 Ansible 那种 ssh style,
被管理的机器只要能被 ssh 就行.
python 用来监控和管理 process 的最好工具是什么? supervisord?
还有, 可以用 python 的某些个 module/lib 来完全代替linux... 阅读全帖 |
|
x****d
发帖数: 1766 | 2 ØMQ (also spelled ZeroMQ, 0MQ or ZMQ) is a high-performance
asynchronous messaging library aimed at use in scalable distributed or
concurrent applications.
Apache ActiveMQ is an open source message broker written in Java together
with a full Java Message Service (JMS) client.
RabbitMQ is open source message broker software (sometimes called message-
oriented middleware) that implements the Advanced Message Queuing Protocol (
AMQP). The RabbitMQ server is written in the Erlang programming la... 阅读全帖 |
|
|
x****d
发帖数: 1766 | 4 ZeroMQ真的很不一样，一般说的mq应该不能把zeromq包括进去。 |
|
d*******r
发帖数: 3299 | 5 二爷你赢了... 不要为自己辩护了, 你是真 Python 黑 :D
顺风问个问题.
https://spark.apache.org/docs/latest/streaming-programming-guide.html
看了眼 spark streaming 的 ingesting methods
Data can be ingested from many sources like Kafka, Flume, Twitter, ZeroMQ,
Kinesis or plain old TCP sockets and be processed using complex algorithms...
如果想做最低 latency 的 processing，直接从网线到memory到Spark，是应该用哪个
ingesting method 呢？
ZeroMQ 或者 干脆直接 TCP sockets? |
|
l*****r
发帖数: 687 | 6 给你个栗子
Second, if we don't have point-to-point, how far can we trust APs today?
Well, if you go to a Starbucks in the US and try the ZeroMQ "Hello World"
example using two laptops connected via the free WiFi, you'll find they
cannot connect. Why? Well, the answer is in the name: "attwifi". AT&T is a
good old incumbent telco that hates WiFi and presumably provides the service
cheaply to Starbucks and others so that independents can't get into the
market. But any access point you buy will support c... 阅读全帖 |
|
k********o
发帖数: 236 | 7 Java backend Developer
Job Description
Development of network facing interfaces and value-added processing of
collected data. Feature and platform work including fast methods of data
movement for streaming as well as other traditional models of data transfer.
Design, code, test, and support such functions for both waterfall and
Agile projects.
Required Skill Set
Java, mutlithreading, Perl, Shell, XML, XSLT, JSON, secure file transfer (e.
g., SSH, SCP), Linux, network interfaces
Desired Skil... 阅读全帖 |
|
k********o
发帖数: 236 | 8 【 以下文字转载自 NewJersey 讨论区 】
发信人: kittyhello (HelloKitty), 信区: NewJersey
标 题: Full time: Java Developer
发信站: BBS 未名空间站 (Tue Feb 10 14:35:16 2015, 美东)
Java backend Developer
Job Description
Development of network facing interfaces and value-added processing of
collected data. Feature and platform work including fast methods of data
movement for streaming as well as other traditional models of data transfer.
Design, code, test, and support such functions for both waterfall and
Agile projects.
R... 阅读全帖 |
|
r****t
发帖数: 10904 | 9 message passing is not suitable to passing large data. but zeromq or
activemq have some use if data is not ery large. |
|
w**z
发帖数: 8232 | 10 activemq, rabbitmq, zeromq, 我们用rabbitmq, 量大有问题。性能下降很快。Kafka
has much big throughput . |
|
k********o
发帖数: 236 | 11 【 以下文字转载自 NewJersey 讨论区 】
发信人: kittyhello (HelloKitty), 信区: NewJersey
标 题: Full time: Java Developer
发信站: BBS 未名空间站 (Tue Feb 10 14:35:16 2015, 美东)
Java backend Developer
Job Description
Development of network facing interfaces and value-added processing of
collected data. Feature and platform work including fast methods of data
movement for streaming as well as other traditional models of data transfer.
Design, code, test, and support such functions for both waterfall and
Agile projects.
R... 阅读全帖 |
|
s*******u
发帖数: 692 | 12 有经验的也没戏
某家一个 号称11年经验的女的， 问messaing queue 的设计
结果一聊， 她不知道 aws simple queue, 也不知道什么 rabbitmq, zeromq 连
activemq 也不熟悉
都不知道怎么跟她聊天了。 |
|
j*****i
发帖数: 7 | 13 最近组内找人，个人感觉待遇很不错，适合有经验的，至少3~4年吧，要求Java要熟练
，做大数据相关，用到很多热门的技术，比如Hadoop/Spark/Storm/Kafka，在上海算是
比较有竞争力的，有意者可帮忙内推，以下是组内职位的JD，如果对其他职位比较感兴
趣，可以自己去eBay career找，应届或实习无法内推。我会每天检查邮件，无论是否
成功都会在在一周内给您回复，请有意者把简历发到[email protected]
/* */。ebay
没有冷冻期，大家可尽力尝试。
Position Title
MTS 2 Software Engineer
Description
This position is located in eBay Center of Excellence in Shanghai in the
Engineering team of eBay Cloud Services organization in eBay. This team is
focusing on engineering design and implementation t... 阅读全帖 |
|
j*****i
发帖数: 7 | 14 最近组内找人，个人感觉待遇很不错，适合有经验的，至少3~4年吧，要求Java要熟练
，做大数据相关，用到很多热门的技术，比如Hadoop/Spark/Storm/Kafka，在上海算是
比较有竞争力的，有意者可帮忙内推，以下是组内职位的JD，如果对其他职位比较感兴
趣，可以自己去eBay career找，应届或实习无法内推。我会每天检查邮件，无论是否
成功都会在在一周内给您回复，请有意者把简历发到[email protected]
/* */。ebay
没有冷冻期，大家可尽力尝试。
Position Title
MTS 2 Software Engineer
Description
This position is located in eBay Center of Excellence in Shanghai in the
Engineering team of eBay Cloud Services organization in eBay. This team is
focusing on engineering design and implementation t... 阅读全帖 |
|
H**********5
发帖数: 2012 | 15 如题。以下这些东东，哪些目前在就业市场更受关注，面试时更可能的被问的详细?
Mongodb
Json
Restful API
Avro
Kafka
Oozie
Azure
Zookeeper
Protobuf
HDFS
Hadoop
Yarn
calsh
Cassandra
RabbitMq
ActiveMq
ZeroMq |
|
m********u
发帖数: 3942 | 16 最好有绿卡~ 有兴趣请站内或者邮件 [email protected]
Job Descriptions:
Our client is looking for a motivated and talented java developer to join
our engineering team to develop the next generation programmatic advertising
system. The candidate will play a key role in the design & development of
a product suite that contributes to the evolution of the programmatic
advertising industry and his/her work will help to differentiate company
from its competitors. Prior experience in programmatic advertising or re... 阅读全帖 |
|
m********u
发帖数: 3942 | 17 最好有绿卡~ 有兴趣请站内或者邮件 [email protected]
Job Descriptions:
Our client is looking for a motivated and talented java developer to join
our engineering team to develop the next generation programmatic advertising
system. The candidate will play a key role in the design & development of
a product suite that contributes to the evolution of the programmatic
advertising industry and his/her work will help to differentiate company
from its competitors. Prior experience in programmatic advertising or re... 阅读全帖 |
|
发帖数: 1 | 18 国内顶级私募基金诚聘英才
本公司是一家专业从事低延迟程序化交易的私募，总部位于北京地标性建筑内。公
司核心成员均毕业于斯坦福、清华、北大、人大、科大等海内外著名大学数学或计算机
相关专业，并曾就职于Morgan Stanley、IMC等世界一流程序化交易机构。我们致力于
通过深度观察分析市场微观数据来研究市场规律，并利用数学、统计、机器学习等方法
开展程序化交易，涉猎期货、期权、股票等多个国内外市场。公司自成立以来，依靠团
队强大的研发平台和专业的开发能力取得了持续多年的优异业绩（主要策略Sharpe
Ratio高于30）
公司以顶尖人才为第一生产力，坚持扁平化管理，遵循平等互助原则，使每位员工
在团队中均承担重要角色，并能在轻松有趣的工作氛围中尽情发挥个人才能。除了极富
竞争力的薪酬体系和标准的五险一金外，公司提供的福利如下：
- 免费三餐（聘请了专职的私厨）和snacks
- 地标建筑办公，环境舒适、风景宜人，自有健身房、娱乐休闲区
- 可升降办公桌，可以站着办公（非常重要！）
- 各种节日大餐和节日礼物（iPhone级别）
- 商业补充医疗 + 高端私立医院报销
... 阅读全帖 |
|
k********o
发帖数: 236 | 19 Java backend Developer
Job Description
Development of network facing interfaces and value-added processing of
collected data. Feature and platform work including fast methods of data
movement for streaming as well as other traditional models of data transfer.
Design, code, test, and support such functions for both waterfall and
Agile projects.
Required Skill Set
Java, mutlithreading, Perl, Shell, XML, XSLT, JSON, secure file transfer (e.
g., SSH, SCP), Linux, network interfaces
Desired Skil... 阅读全帖 |
|
k********o
发帖数: 236 | 20 【 以下文字转载自 NewJersey 讨论区 】
发信人: kittyhello (HelloKitty), 信区: NewJersey
标 题: Full time: Java Developer
发信站: BBS 未名空间站 (Tue Feb 10 14:35:16 2015, 美东)
Java backend Developer
Job Description
Development of network facing interfaces and value-added processing of
collected data. Feature and platform work including fast methods of data
movement for streaming as well as other traditional models of data transfer.
Design, code, test, and support such functions for both waterfall and
Agile projects.
R... 阅读全帖 |
|
m********u
发帖数: 3942 | 21 【 以下文字转载自 JobHunting 讨论区 】
发信人: missingyou (miss), 信区: JobHunting
标 题: Fullstack Engineer 三番
发信站: BBS 未名空间站 (Wed Jun 28 02:43:20 2017, 美东)
有兴趣请站内或者邮件 [email protected]
Job Descriptions:
Our client is looking for a motivated and talented java developer to join
our engineering team to develop the next generation programmatic advertising
system. The candidate will play a key role in the design & development of
a product suite that contributes to the evolution of the programmatic
advertising ... 阅读全帖 |
|
m********u
发帖数: 3942 | 22 【 以下文字转载自 JobHunting 讨论区 】
发信人: missingyou (miss), 信区: JobHunting
标 题: Java Developer 三番
发信站: BBS 未名空间站 (Wed Jun 28 02:45:12 2017, 美东)
有兴趣请站内或者邮件 [email protected]
Job Descriptions:
Our client is looking for a motivated and talented java developer to join
our engineering team to develop the next generation programmatic advertising
system. The candidate will play a key role in the design & development of
a product suite that contributes to the evolution of the programmatic
advertising indu... 阅读全帖 |
|
x****d
发帖数: 1766 | 23 What are you guys reading or suggest reading/learning for distributed
computing in java? I don't mean big data thing, but JEE.
Solr/Hadoop is using zookeeper. So I guess Zookeeper is a good starting
point. I looked into it a bit, didn't catch much info.
If I have to do something similar, what other options I have other than
using zookeeper to keep my config files in sync? I can think of using JMS/MQ
. Somebody says a look in ZEROmq, but that is something like protocal, to
build something useful,... 阅读全帖 |
|
x****d
发帖数: 1766 | 24 webservice is not exactly the same as messaging, jms is messaging, messaging
can do publish/sub and other messaging patterns, where webservice need to
be twisted to implement.
jms is not a protocol, it is a java thing. If you want cross language, you
might need protocol base solution. Like amqp, mqtt.
if you want to define your own protocol and dont want to program tcp/sockets
, you can try zeromq.
JMS brokers nowadays is not language dependent. So you can use activemq or
sonicmq or rabbitmq, as... 阅读全帖 |
|
d***q
发帖数: 1119 | 25 python （估计，ruby, 或者其他动态语言也可能适用）
的 function call是个很重型的动作，vm又比较差劲，没法很好地Inline 来避免开销，
+ 类型的检查，导致如果发生大量函数调用时就会很慢。于是不少人搞出了一些
例如 cython,swig之类的东西，本质就是要绕过pvm， 降低函数调用的损失，来达到加
速的效果。
GIL 这个可以用多进程来部分解决这个问题，也有不少应用的例子，例如web的
前端用nginx, 后端起若干个进程，或者用zeromq之类来进行进程间的通信，都有真实
的例子，所以gil 现在反而不是太敏感。 |
|
x****d
发帖数: 1766 | 26 我不是这方面专家。这个路还很长，不是说没具体方案。你去看commotion和zeromq吧
，看完你至少会得到些启发。其实还有很多方案，考虑家电上网春天还有几年，摩尔定
理推算，家电上有一个比现在手机强两三倍计算能力的东东不是很不靠谱吧？
让我用我懂的技术做，我都能做出来，用embendded的activeMQ都行。不就是messaging
么？
我家冰箱电视还要给中宣部发布信息？给cia，fbi监控？ WTF！ |
|
d****i
发帖数: 4809 | 27 看到他们的readme, 把clojure写的storm批了一顿:
Storm is a wonderful product, but it is developed in Clojure,which is not a
popular language. And in our company, few guys can fix the bug of Storm,
which we already found in zeromq, zookeeper, performance,etc.
So, we decided to implement JStorm.
JStorm is more stable than Storm;
JStorm is faster than Storm;
JStorm provides several new and very useful features. |
|
d*******r
发帖数: 3299 | 28 多谢指教，我再琢磨琢磨。
感觉Kafka 现在是最 hot 的message queue了. 问个菜的问题，它主要好处就是
performance 不错，加上 using persistence by default? 主要受欢迎点在哪里？难
道就是因为用 Scala 写的?
我还记得以前都是一堆 Rabbitmq 和 zeromq 来着. |
|
d*******r
发帖数: 3299 | 29 这些都太底层了，
zeromq 有核心离开了，搞了个 www.crossroads.io
vert.x 是 build 在 netty 上的，直接用更方便 |
|
M**A
发帖数: 78 | 30 谢谢魏老师的机器。
连接不上 http://54.197.82.110
请问5MM/sec 是指处理5,000,000transactions/sec, 还是处理5,000,000requests/sec？
最近我正在测试3种web server的性能。
基本配置： 8 core 64-bit CPU, 16G RAM, MySQL
(1) Tomcat 可以处理 6,000 HTTP requests/sec
http://tomcat.apache.org/
(2) SIMPLE 可以处理 8,000 HTTP requests/sec
http://www.simpleframework.org/
(3) vert.x 可以处理10,000 HTTP requests/sec
http://vertx.io/
如果需要SSL加密和解密，Tomcat只能处理 250 HTTPS requests/sec， 发现
bottleneck在每建立SSL连接需要3方hand-shake.
下面链接提到ESPN处理10K requests/sec的方案。
http://highscalabi... 阅读全帖 |
|
l*****t
发帖数: 2019 | 31 kafka就是log shipping，不算mq。只不过有个pub-sub的架势。如果是low latency的
mq，估计考虑一下zeromq，kafka的paper上好像说zero是rabbit高一倍的throughput。
你看看、试试吧。 |
|
d****i
发帖数: 4809 | 32 MQ为什么会有瓶颈？我觉得传统而成熟的ActiveMQ, ZeroMQ等等都很好啊。 |
|
|
x****d
发帖数: 1766 | 34 你用zeromq做个hello world，再用ActiveMQ做一个，RabbitMQ做一个，就清楚得很了
。 |
|
s********k
发帖数: 6180 | 35 这样看来用zeroMQ成本更高啊？开发需要专门做而不是配置下就行，好处在哪里呢？
(
and
failover
major |
|
g*****g
发帖数: 34805 | 36 zeroMQ is a chopping knife. |
|
p*****2
发帖数: 21240 | 37
zeromq如何？storm最早用的这个。不然就用AKKA？ |
|
z*****u
发帖数: 62 | 38 kafka 还是解决scalability 问题的吧？这种单一master单一replica 还是
performance的问题吧？open source的应该也挺多的。zeromq啥的据说都不错。 |
|
h****e
发帖数: 2125 | 39 比较light weight，但是又提供persistence的message queue？看来看去好像只有
RabbitMQ，还是比较慢哪。 |
|
|
|
p*u
发帖数: 2454 | 42 Kafka has better throughput but higher latency |
|
h****e
发帖数: 2125 | 43 nanomsg也不提供persistence吧。 |
|
|
|
|
h**l
发帖数: 168 | 47 yeah, that is true. ZeroMQ author eventually joined Google to be a senior
software engineer maybe because he found it is his only way to make more
money. :) |
|
p*u
发帖数: 2454 | 48 ZeroMQ写的很差，很多问题没法上production。那个作者傻叉一个还牛逼哄哄的，结果
被踢出project。后来他自己fork了一下用C重写，过了没多久也弃了，很不负责任。 |
|
发帖数: 1 | 49 国内顶级私募基金诚聘英才
本公司是一家专业从事低延迟程序化交易的私募，总部位于北京地标性建筑内。公
司核心成员均毕业于斯坦福、清华、北大、人大、科大等海内外著名大学数学或计算机
相关专业，并曾就职于Morgan Stanley、IMC等世界一流程序化交易机构。我们致力于
通过深度观察分析市场微观数据来研究市场规律，并利用数学、统计、机器学习等方法
开展程序化交易，涉猎期货、期权、股票等多个国内外市场。公司自成立以来，依靠团
队强大的研发平台和专业的开发能力取得了持续多年的优异业绩（主要策略Sharpe
Ratio高于30
公司以顶尖人才为第一生产力，坚持扁平化管理，遵循平等互助原则，使每位员工
在团队中均承担重要角色，并能在轻松有趣的工作氛围中尽情发挥个人才能。除了极富
竞争力的薪酬体系和标准的五险一金外，公司提供的福利如下：
- 免费三餐（聘请了专职的私厨）和snacks
- 地标建筑办公，环境舒适、风景宜人，自有健身房、娱乐休闲区
- 可升降办公桌，可以站着办公（非常重要！）
- 各种节日大餐和节日礼物（iPhone级别）
- 商业补充医疗 + 高端私立医院报销
-... 阅读全帖 |
|
